Incident Response and Secure Development Habits

Plan the first hour before you need it

When an LLM incident happens (data leaked in an answer, a tool abused, a huge bill, a harmful output posted publicly), the response has the usual shape. Contain: have switches to disable a tool, a feature or the whole assistant, block a user, rotate exposed credentials and revert a prompt or model version. Investigate: use audit logs to see what was asked, what was retrieved, which tools ran and with whose permissions. Notify: follow your data-breach and customer-communication obligations. Fix and test: close the weakness (usually a missing control, not a better prompt), add the attack to the regression suite, and verify. Learn: run a blameless review and update the threat model. As habits: do a threat model for each new capability, review AI features in code review with a security checklist, keep secrets out of prompts, follow least privilege, keep dependencies pinned, and train the team on prompt injection. Align with the standards and laws that apply to you.

An LLM incident runbook

Fill in owners and switches for your own system.

1. CONTAIN   disable feature flag "assistant-tools"; block user/IP; rotate exposed keys; pin previous prompt+model
2. PRESERVE  snapshot audit logs, prompts, retrieved chunks, tool calls (access-controlled)
3. ASSESS    what data/actions were reachable? which users affected? was it exfiltrated or only exposed?
4. NOTIFY    security lead, legal/privacy, affected customers as required by contract and law
5. FIX       add the missing control (permission, filter in retrieval, approval, cap); not just a prompt tweak
6. TEST      add the attack to the regression suite; re-run the full attack suite
7. LEARN     blameless review; update threat model and runbook

Practise the kill switch

Run a drill: disable a tool in staging and confirm the app degrades gracefully.

Quick check: After an incident, what is usually the right kind of fix?

• Adding the missing control (permissions, validation, limits) and a regression test
• Only rewording the system prompt
• Hoping it does not recur
• Deleting the logs