Data Poisoning and Untrusted Knowledge Sources

Poisoned knowledge gives poisoned answers

Anything the model learns from or retrieves can be poisoned. In RAG, an attacker who can add or edit a document in your index (a wiki page, a support article, an uploaded file, a web page you crawl) can plant false facts or hidden instructions that are later retrieved for other users. In fine-tuning, bad examples can embed wrong behaviour or backdoors triggered by a special phrase. In agent memory, injected "facts" can persist across sessions. Defences: control who can add content and require review for sensitive sources; keep provenance (source, author, date) on every chunk and show citations; sanitise content on ingestion (strip hidden text and scripts); version and scan training data; test the finished system with questions whose answers you know; and let users inspect and delete stored memory. Treat your knowledge base as part of your attack surface.

A poisoning-resistant ingestion checklist

Controls at the point where content enters the knowledge base.

[ ] who can add or edit sources? least privilege + review for sensitive collections
[ ] provenance stored per chunk: source URL/file, author, ingestion date, content hash
[ ] hidden text, scripts and metadata stripped before indexing
[ ] untrusted web content kept in a separate, lower-trust index; answers cite sources
[ ] fine-tuning data versioned, scanned, and spot-checked for odd trigger phrases
[ ] canary questions with known answers run after every index update
[ ] users can inspect and delete stored memory; entries have an owner and an expiry

Run canary questions after updates

Questions with known answers reveal a poisoned or broken index quickly.

Quick check: Why store provenance (source, author, date) with each chunk?

• To make chunks larger
• To trace, cite and remove poisoned or outdated content
• Because models require it
• There is no benefit