Retries and backoff for LLM calls

Calls to model APIs fail more often than most internal services. Rate limits, overloaded servers and network timeouts all happen. A good retry policy turns most of these into non-events.

Retry only what can change

Retry errors that are likely to go away on their own:

• rate limit responses (429)
• server errors (5xx)
• timeouts and dropped connections

Do not retry errors that will fail the same way again, such as an invalid request, a bad API key or a prompt that is too long. Retrying those wastes time and money.

Wait longer each time, with jitter

Exponential backoff doubles the wait after each failure. Jitter adds randomness so that many clients do not all retry at the same instant.

import random
import time

def call_with_retry(fn, attempts=5, base=1.0, cap=30.0):
    for attempt in range(attempts):
        try:
            return fn()
        except TransientError:
            if attempt == attempts - 1:
                raise
            delay = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, delay))

If the server sends a retry-after hint, respect it.

Set a limit

Always cap the number of attempts and the total time. An agent that retries forever looks like it is working while it is actually stuck.

Retries and side effects

A retry of a call with side effects needs the idempotency handling from the previous post. Otherwise a retry can do the work twice.