How Coding Agents Fail

Plausible is not correct

Common failure modes: wrong understanding (solves a different problem than you meant); hallucinated APIs or packages (calls functions that do not exist, or invents a dependency name, which attackers can register); overreach (changes unrelated code, reformats files, "improves" things you did not ask for); faking success (special-cases the tests, weakens assertions, claims "all tests pass" without running them); getting stuck in loops (repeating the same failing action); stale or lost context (editing from an outdated view of a file, forgetting earlier decisions); unsafe actions (destructive commands, leaking secrets, following instructions hidden in a web page or file: prompt injection); and subtle bugs that tests do not cover. Most of these are detectable by the same mechanisms: independent checks, small diffs and human review of the parts that matter.

Check the work, bound the run

Independent checks, loop detection, patch gates and secret scans catch most agent failures before they matter.

A review checklist for agent diffs

Skim for these before reading line by line.

[ ] does the diff solve the stated task, and ONLY that task? (scope)
[ ] any test removed, skipped or loosened? any special-casing of test inputs?
[ ] new imports / dependencies: do they exist, are they intended?
[ ] error paths and edge cases handled (empty, None, large, concurrent)?
[ ] secrets, tokens, URLs or credentials added? config/CI/permission files touched?
[ ] unrelated reformatting or renames hiding real changes?
[ ] did I see test output, or only the agent saying "tests pass"?

Verify new dependencies

Check that any package an agent adds exists, is maintained, and is the one you intended.

Quick check: Why can a hallucinated package name be dangerous?

• It is always blocked by Python
• It makes tests faster
• An attacker may publish a malicious package under that name
• It cannot be installed