Rate limits are not the enemy. They are a contract.
When an LLM API returns 429, it is saying: you are asking for more than we agreed. The wrong response is to immediately ask again. The right response is to back off, queue the work, and ask again when the contract allows it.
Most retry implementations get this backwards. They treat 429 like a transient glitch — something to power through with enough retries. The result is retry storms that make the problem worse, cascading failures that take down more than just the LLM feature, and bills that reflect the cost of all those failed attempts.
Here is what actually works.
Why naive retries fail
The typical implementation looks something like this:
async function callLLM(prompt: string, retries = 3): Promise<string> {
try {
return await llm.generate(prompt);
} catch (err) {
if (retries > 0) return callLLM(prompt, retries - 1);
throw err;
}
}This has four problems:
No delay between retries. If the first call hit a rate limit, the retry hits the same limit immediately. You are not solving the problem — you are amplifying it.
No distinction between error types. A rate limit (429) and a model overload (503) and a bad request (400) all look the same to this code. A 400 will never succeed on retry. A 429 needs time. A 503 needs a different amount of time than a 429. Treating them identically is wrong for all of them.
No circuit breaker. If the API is consistently rate limiting you, 10 sequential callers all retrying independently will make it worse for everyone, including themselves.
Retry cost is invisible. Failed LLM calls that are rejected before processing cost nothing in tokens. But the HTTPS round trips, the time, and the retry overhead still count toward your throughput budget and your system’s apparent load.
The Retry-After header is the API telling you what to do
When Anthropic (and most LLM providers) return a 429, they include a Retry-After header. This header says, in plain numbers: wait this many seconds before trying again.
The most reliable retry strategy is also the simplest: read that header and wait that long.
async function callWithRetry(prompt: string): Promise<string> {
for (let attempt = 0; attempt < 4; attempt++) {
const res = await llm.generate(prompt);
if (res.status === 429) {
const retryAfter = parseInt(res.headers.get('Retry-After') ?? '60', 10);
const jitter = Math.random() * 5; // spread concurrent retries
await sleep((retryAfter + jitter) * 1000);
continue;
}
if (!res.ok) throw new Error(`LLM error: ${res.status}`);
return await res.json();
}
throw new Error('Rate limit retries exhausted — job queued for later');
}The jitter matters. If 20 concurrent requests all hit a rate limit and all retry after exactly 60 seconds, you get a synchronized stampede that immediately triggers another rate limit. Random jitter between 0–5 seconds spreads the retries across time and prevents the thundering herd.
Exponential backoff for errors that aren’t rate limits
For 5xx errors (model overloaded, service unavailable), the API isn’t giving you a specific wait time — it’s saying it’s under load. Exponential backoff with a cap is the right call here:
const BASE_DELAY_MS = 1_000;
const MAX_DELAY_MS = 32_000;
function backoffMs(attempt: number): number {
const exp = Math.min(BASE_DELAY_MS * 2 ** attempt, MAX_DELAY_MS);
return exp + Math.random() * 1_000; // jitter
}Attempt 0: ~1s. Attempt 1: ~2s. Attempt 2: ~4s. Attempt 3: ~8s. Capped at 32s.
Do not retry 4xx errors that aren’t 429. A 400 bad request, a 401 auth error, a 413 payload too large — these will not fix themselves with time. Retrying them wastes time and obscures the real error.
Circuit breakers for sustained failures
If you are hitting rate limits continuously, something is wrong at the architectural level — not the retry level. A circuit breaker detects this and stops hammering:
class LLMCircuitBreaker {
private failures = 0;
private openUntil = 0;
private readonly threshold = 5; // open after 5 consecutive failures
private readonly resetMs = 60_000; // try again after 60s
isOpen(): boolean {
return this.failures >= this.threshold && Date.now() < this.openUntil;
}
recordSuccess() { this.failures = 0; }
recordFailure() {
this.failures++;
if (this.failures >= this.threshold) {
this.openUntil = Date.now() + this.resetMs;
}
}
}When the circuit is open, jobs go to a queue instead of being attempted. The queue drains once the circuit closes. This is how you prevent a rate limit storm from cascading into broader system failures.
Queue the work, don’t block the caller
For non-latency-critical LLM work — content generation, background enrichment, scheduled posts — the right pattern is not to retry in-line at all. Put the job in a queue and let a worker drain it at a controlled rate:
[API request] → [Redis queue] → [worker, rate-controlled] → [LLM API]The worker runs at a rate you control. It knows your token budget. It backs off when it hits limits. The caller gets an immediate response (“queued”) and isn’t blocked waiting for a generation that might take 65 seconds to succeed after a rate limit.
This is how fabric-social handles all non-interactive LLM work: validate fast, queue, drain at a controlled rate. The user experience is never blocked by a rate limit.
The strategy map
| Error | Strategy |
|---|---|
| 429 with Retry-After | Wait exactly Retry-After + jitter, retry |
| 429 without Retry-After | Exponential backoff starting at 5s |
| 503 / 529 (overloaded) | Exponential backoff, cap at 32s |
| 5 consecutive failures | Circuit breaker, queue jobs for 60s |
| Non-interactive work | Queue always, never block caller |
| 400 / 401 / 413 | Do not retry — fix the request |
What to actually monitor
- Rate limit hit rate — if this is > 5%, you need to revisit your concurrency or your queue drain rate
- Retry success rate — what % of rate-limited requests eventually succeed? Low rate = architectural problem
- Circuit breaker trips — each trip means you had sustained failure; investigate the cause
- P99 latency including retries — the tail latency after retries tells you what users actually experience
Rate limits are the API telling you something about how you’re using it. Good retry strategy is about listening to that signal and responding appropriately — not about finding ways to ignore it.
Frequently Asked Questions
What is the difference between a 429 and 503 error from an LLM API?
A 429 error means you have exceeded your rate limit quota and must wait for your allowance to reset — the Retry-After header tells you exactly how long. A 503 error means the model or service is temporarily overloaded, which is a different condition that typically requires a shorter but still delayed retry, not an immediate one.
Should you retry a 400 error from an LLM API?
No. A 400 Bad Request error indicates a malformed or invalid request that the server rejected — retrying the exact same request will always fail. Only errors like 429 and 503 are candidates for retry logic.
What causes a retry storm when calling LLM APIs?
A retry storm occurs when multiple callers independently retry failed requests with no delay or coordination, multiplying the total request volume against an API that is already rate limiting. Adding per-retry delays, jitter, and a shared circuit breaker prevents independent retries from compounding the problem.
How should you implement rate limit retry logic for Anthropic or OpenAI APIs in TypeScript?
Read the Retry-After header from the 429 response and wait that exact duration before retrying. Add exponential backoff with jitter for cases where Retry-After is absent, implement a circuit breaker to halt retries when failures are sustained, and never retry 400-class errors. Distinguish error types explicitly rather than catching all errors with a single retry path.
