Staged Rollouts, Shadow Mode and Rollback

Prove it on a little traffic first

Even after passing offline evaluation, a change may behave differently on real traffic. Release in stages: shadow mode (run the new version on live requests in parallel, log its outputs, show users the old version, compare offline); canary (route 1 to 5% of traffic to the new version and watch quality, latency, cost, error and feedback metrics); then ramp up (25%, 50%, 100%) if the numbers hold. Keep the old version deployed and a one-step rollback, controlled by configuration or a feature flag rather than a redeploy. For user-facing changes, run an A/B test with a clear success metric and enough traffic to detect the difference you care about; remember the noise lessons: small gaps need many samples. Define in advance what result triggers a rollback, and who decides. Communicate changes to support and product teams so they know when behaviour shifts.

A rollout plan

Each stage has a duration, a watch list and a rollback rule. Numbers are examples.

stage        traffic   duration   watch                                   rollback if
shadow       0% (logged) 2 days    judge score vs old, cost, latency        new version loses by > noise margin
canary       2%        1 day      validity rate, p95, thumbs-down, cost    p95 +30% or validity < 99%
ramp 1       25%       2 days     same + category breakdown                any category -5 points
ramp 2       50%       2 days     same                                     any critical safety event
full         100%      keep old   dashboards + alerts                      one-step switch to previous release

Decide rollback rules in advance

Write down what result triggers a rollback and who decides.

Quick check: What does shadow mode do?

• Deletes the old version
• Shows the new version to everyone immediately
• Runs the new version on live requests without showing its output to users, for comparison
• Disables logging