Failure Handling in Layers

Plan each failure, then test it

Typical failures and a layered response: transient provider errors and rate limits → retry with exponential backoff and jitter, within a total time budget; provider outage → fail over to another model or provider (keep prompts portable and test the fallback regularly), or a cached or simpler answer; slow responses → timeouts, streaming, hedging for idempotent calls; invalid output → bounded repair loop, then safe default or human queue; retrieval empty or weak → say "I could not find this" rather than guess; tool errors → return clear error text to the model, limit retries, never repeat side-effecting actions blindly; runaway loops → step, token, time and cost caps; overload → queue, shed low-priority work, return a clear "busy" message. Add a circuit breaker to stop calling a failing dependency for a short time. Then run failure drills in staging: disable the provider, inject slow responses and bad outputs, and confirm users get a sensible experience.

Failure modes and responses

Fill in your own system; test each row in staging.

Failure                         Layered response
rate limit / 5xx / timeout       retry with backoff+jitter inside a time budget -> fallback model -> cached/simple answer
provider outage                 circuit breaker -> secondary provider (portable prompts) -> graceful message
invalid structured output       bounded repair (<= 2) -> safe default or human queue; count the failure
empty or weak retrieval         "I could not find this in our documents" + offer a human; do not guess
tool error                      clear error text to the model; capped retries; no blind repeat of side effects
runaway agent loop              step/token/time/cost caps -> stop, report what was tried
overload                        queue, shed low priority, "busy, try again" with retry hints

Run a failure drill each quarter

Disable the provider in staging and confirm users get a sensible fallback.

Quick check: What is the safest response when retrieval returns nothing relevant?

• Invent a plausible answer
• Say the answer was not found, and offer a human
• Return an empty page
• Crash the app