Release Gates and CI for LLM Changes

Run the evaluation on every change

A prompt edit is a code change and deserves the same safety net. In continuous integration, run: fast unit tests with fake models for the deterministic code (parsing, routing, policy), the evaluation set against the candidate bundle, the security attack suite, and cost and latency checks. Then apply a gate with explicit rules, for example: overall pass rate must not drop by more than the noise margin and must stay above the target; no category may fall by more than a set amount; zero critical safety failures; p95 latency and average cost within budget. Show the diff of flipped cases to the reviewer. Full evaluations with real models cost money and time, so run a small smoke subset on every commit and the full set before merge or release. When the gate fails, the fix is investigation, not loosening the threshold.

A release gate expressed as rules

Thresholds are examples. The point is that the decision is mechanical and written down.

GATE (all must hold, else block the merge)
1. unit tests (fake models): all pass
2. eval pass rate >= 0.92 AND not lower than baseline by more than 0.02 (the noise margin)
3. no category drops by more than 0.05; Hindi/Hinglish category checked separately
4. attack suite: 0 critical successes; <= 2% minor
5. p95 latency <= 3.0 s; mean cost per request <= 1.1 x baseline
6. reviewer has seen the list of flipped cases (fixed vs broken)

If the gate fails: investigate the flipped cases. Do not just lower the threshold.

Use a small smoke set per commit

Fast checks on every commit and the full evaluation before merge balance speed and cost.

Quick check: What should you do when the release gate fails?

• Delete the failing cases
• Lower the threshold
• Skip the gate once
• Investigate the flipped cases and fix the cause