Safe LLM Retries: 429s, Broken Streams and Duplicate Tool Actions
Build safe LLM retry rules for rate limits, broken streams and tool actions without causing retry storms or duplicate side effects.
On this page
A retry sounds simple: wait, then send the request again. That is safe only when the last attempt clearly failed before it changed anything.
LLM workflows are harder. A provider can reject a request with a 429. A stream can break after the model has produced part of an answer. A tool can finish its work while the client times out. If every failure leads to another attempt, one task can create two tickets, send two emails or charge twice.
The safe rule is simple: retry only when you know the last attempt did not complete. When the result is unknown, check state before acting again.
Why LLM retries become workflow bugs
A normal HTTP policy sees a status code. An agent workflow has more states: model response, streamed events, tool calls, side effects and saved results. Each layer can fail at a different time.
Use one place to record the request, attempt, model route and final state. The LLM Gateway can centralize routing, fallback and usage records. The service that owns a tool must still protect its own writes.

The retry path should change with the failure: wait for rate limits, restart safe reads, and check state before repeating a write.
Match the response to the failure
- 429 rate limit: Honor
Retry-Afterwhen the provider sends it. Add exponential backoff and random jitter. Stop after a small attempt budget. A spend-cap error may not clear with time, so read the error body before looping. - 500, 502, 503 or overload: Retry a small number of times. Open a circuit breaker when failures keep rising. This protects the provider and your queue.
- Broken stream: Treat the result as incomplete. Save only complete events. Do not join two partial answers and call them one response. Restart only when the task is safe to replay.
- Tool timeout: Treat the outcome as unknown. Look up the action by its stable ID before sending it again.
The LLM Gateway documentation covers streaming, fallback and error metrics. Those records help the team see whether a request failed before or after a route changed.
Put a budget around every attempt
Set a budget for each request:
- Maximum attempts for one user request.
- Maximum wall-clock time.
- Maximum token or cost budget across all routes.
Then define which fallback models may handle the task. A cheaper model may be fine for tagging but not for a structured tool plan. The guide to reading LLM model metadata shows why one fallback list should not serve every workload.
Keep one action ID across every attempt
The safest tool retry returns a saved result when it sees the same key again.
async function runOnce(taskId: string) {
const prior = await ledger.get(taskId);
if (prior) return prior;
const result = await callTool({
idempotencyKey: taskId
});
await ledger.save(taskId, result);
return result;
} This example is small on purpose. A production design must also handle a crash between the tool action and the ledger save. The key idea stays the same: one logical action keeps one stable ID across every attempt. Stripe’s idempotency guide uses the same pattern for API writes.

A duplicate action key should return the saved result instead of running the tool twice.
Test the unknown state
Test the failure paths that create uncertainty:
- Break a stream after the first tool call.
- Return a timeout after the tool finishes.
- Send the same idempotency key twice.
- Crash after the action but before the ledger save.
- Fail both the primary model and the fallback.
- Exhaust the retry and cost budgets.
Confirm that logs show the original request, every attempt, each route and one final state. The article on what an LLM gateway controls explains why tokens, routes, fallbacks and audit records belong in the same operating view.
Define the final state before you retry
A retry policy needs one clear finish line. Use states such as pending, running, succeeded, failed and unknown. Only the service that owns the work should move an action to succeeded. A client timeout must not change that result.
For every attempt, log the request ID, action ID, model route, attempt number, start time, end time and final state. Also record whether a tool was requested and whether it completed. These fields let an on-call engineer answer the key question: did the last attempt fail, finish or become unknown?
When the state is unknown, pause the retry. Read the action record or ask the tool for status. If the tool has no lookup method, add one before automatic retries. This small control is often more useful than adding another fallback model.
Use the same rules in background jobs and user-facing requests. The wait time may change, but the safety rule should not.
Make retries boring
A safe retry policy is narrow. It knows which failures are temporary, which actions are safe to replay and when to stop. Keep one action ID through the whole flow. Test broken streams and unknown outcomes, not just neat HTTP errors.
Use the gateway for shared model policy and observability. Keep side-effect safety in the service that owns the action. That split makes retries predictable and duplicate work much easier to prevent.
Frequently Asked Questions
When is it safe to retry an LLM request?
Retry when you know the previous attempt did not complete and the failure is temporary. If a stream or tool outcome is unknown, check state before sending another attempt.
How do you prevent duplicate tool actions during retries?
Carry one stable idempotency or action ID through every attempt and have the tool owner return the saved result when that key has already completed.
What should an LLM retry budget include?
Set a maximum attempt count, wall-clock time, and token or cost budget, then limit fallback models to approved routes that can handle the same task.
How should a broken stream be handled?
Treat a broken stream as incomplete, save only complete events, and restart only when the task is safe to replay. Do not combine two partial answers into one result.
Written by
Axiom Studio