LLM API Error Handling: Retries, Rate Limits, Timeouts, and Fallbacks

Overview

Reliable LLM app development starts by treating an API call as a distributed-system operation, not a simple function call. Your application depends on a network, an external service, a selected model, authentication, quota, input limits, output validation, and sometimes downstream tools. Any of those dependencies can fail independently.

A useful error-handling design separates four questions:

1. Can the request succeed if it is attempted again? This determines whether a retry is appropriate.
2. How long can the user or workflow wait? This determines the timeout budget.
3. What is the safest degraded experience? This determines whether to use another model, provider, cached result, queue, or human review.
4. Can the failure be diagnosed later? This determines what you log, measure, and trace.

Do not retry every exception. Authentication failures, invalid request parameters, unsupported features, malformed schemas, and context-length violations generally require a code or input change. Temporary overload, connection interruption, and some server-side failures may be retryable, but the exact behavior depends on the provider and SDK. Treat provider documentation as the final authority for status codes, retry headers, quotas, and error objects.

For applications that require machine-readable results, pair transport-level handling with output validation. The patterns in Structured Output with LLMs are useful when a successful HTTP response still contains unusable or incomplete data.

Checklist by scenario

When the API returns a rate limit

• Read the response status, error body, and any provider-supplied retry or rate-limit headers.
• Determine whether the limit applies to requests, tokens, a project, an account, a model, or another scope.
• Apply exponential backoff with jitter rather than retrying immediately.
• Cap the number of attempts and the total elapsed time.
• Reduce concurrency or place work on a queue if the traffic pattern is sustained.
• Return a clear temporary-failure state when the retry budget is exhausted.

A basic retry policy can be represented as:

for attempt in 0..max_attempts:
    response = call_llm(timeout=remaining_budget)

    if response.success:
        return response

    if not is_transient(response) or attempt == max_attempts:
        return handle_failure(response)

    delay = retry_after_header(response)
    if delay is missing:
        delay = min(max_delay, base_delay * 2 ** attempt)
        delay = add_random_jitter(delay)

    sleep(min(delay, remaining_budget))

Use a retry-after value when the provider supplies one, subject to your own maximum delay and deadline. Jitter prevents many workers from waking and retrying at the same moment.

When a request times out

• Set an explicit connection timeout and an explicit response or read timeout.
• Pass a deadline through each retry so attempts cannot exceed the user-facing budget.
• Cancel or abandon work that no longer matters to the caller.
• Record whether the timeout happened before connection, during response generation, or while reading a streamed response.
• Decide whether a timed-out request is safe to retry, especially if it triggered tool calls or other side effects.

Streaming requires extra care. A connection can succeed while the model stops sending tokens midway through the response. Detect incomplete streams, validate the final message, and avoid treating partial output as a completed answer unless the product explicitly supports that state.

When the provider or model is unavailable

Use a fallback decision tree rather than a long list of automatic retries:

1. Transient and within budget: retry with backoff.
2. Model-specific failure: try an approved equivalent model if it supports the same input, output, tools, and safety requirements.
3. Provider-specific failure: route to another configured provider only if data handling, quality, latency, and contract requirements allow it.
4. Nonessential asynchronous task: queue the job for later processing and expose its status.
5. High-impact or uncertain result: pause and escalate to a human rather than silently substituting a weaker path.

Fallbacks should be explicit in product design. A cheaper or smaller model may change formatting, reasoning quality, context capacity, or tool behavior. Validate the fallback response using the same schema and business rules as the primary response.

When the response is valid but unusable

• Validate required fields, types, length limits, and allowed values.
• Separate refusal, empty output, malformed output, and tool failure in your error taxonomy.
• Retry with a corrected prompt only when the correction is deterministic and the retry budget permits it.
• Do not repeatedly ask the model to repair output without a stopping condition.
• Store a safe user-facing message when no valid result can be produced.

For tool-enabled applications, review the function calling tutorial and distinguish model-generated tool arguments from the tool's actual execution result.

What to double-check

HTTP and SDK error mapping

Build an internal error taxonomy instead of scattering provider-specific conditions throughout the codebase. A reference mapping can look like this:

This table is a design starting point, not a substitute for current provider documentation. SDKs may convert HTTP responses into typed exceptions, retry automatically, or return partial results. Confirm those behaviors before adding your own retry layer, or you may multiply attempts unintentionally.

Observability and privacy

• Assign a request ID and correlation ID that follow the request across queues, model calls, and tools.
• Log provider, model, operation type, attempt number, latency, outcome category, and fallback path.
• Track retry rate, timeout rate, validation failures, queue age, and successful completion after fallback.
• Redact secrets and avoid storing sensitive prompts or outputs unless there is a documented reason and retention policy.
• Keep enough metadata to compare failures without exposing user content.

For broader latency and cost controls, see LLM Cost and Latency Optimization. Reliability work is easier when retries, token usage, and fallback traffic are visible together.

Common mistakes

• Retrying permanent errors: Repeating an invalid request increases load and delays a fix.
• Using fixed delays: Identical workers can create synchronized retry storms.
• Ignoring total deadlines: Several individually reasonable retries can produce an unacceptable user wait.
• Retrying side effects blindly: A timed-out tool call may have succeeded even if its response was lost. Use idempotency keys or reconciliation where supported.
• Failing over without compatibility checks: Another model may not support the same context size, tools, schema, or output quality.
• Returning raw provider errors: Error payloads can expose implementation details and are rarely useful to end users.
• Hiding repeated failures: A generic success response can turn an operational problem into silent data loss.
• Testing only the happy path: Simulate rate limits, slow responses, dropped streams, malformed output, expired credentials, and unavailable fallbacks.

Also review security boundaries when an LLM can call tools or access private data. The AI agent security checklist covers permissions, prompt injection, secrets, and tool access that should remain enforced during fallback execution.

When to revisit

Revisit this LLM API error-handling checklist before seasonal planning cycles, traffic campaigns, or any release expected to change request volume. Review it whenever you add a provider or model, enable streaming or tool use, change prompts or schemas, alter queue concurrency, or introduce a new user-facing latency target.

Schedule a practical review whenever workflows or tools change. Confirm current provider status codes, retry headers, SDK defaults, timeout behavior, quotas, model availability, and fallback compatibility directly against the relevant documentation. These details can change independently of your application code.

At each review, run a failure exercise:

1. Trigger a rate-limit response and verify backoff and concurrency control.
2. Delay the response beyond the deadline and confirm cancellation or safe timeout behavior.
3. Return malformed structured output and verify validation and escalation.
4. Disable the primary model or provider and test the approved fallback.
5. Inspect logs and metrics for request correlation, privacy, and actionable diagnosis.
6. Confirm that users receive an honest, useful status rather than an invented answer.

Finally, keep the retry policy, fallback matrix, and error taxonomy close to the integration code and document who owns updates. A short, tested policy is more valuable than a complex recovery system that nobody can explain during an incident.