# Data Poisoning and Untrusted Knowledge Sources — LLM Application Security

Source: https://www.geekswithgeeks.com/en/llm-security/d-poison

> Control what enters your knowledge base, fine-tuning data and memory.

## Poisoned knowledge gives poisoned answers

Anything the model learns from or retrieves can be **poisoned**. In **RAG**, an attacker who can add or edit a document in your index (a wiki page, a support article, an uploaded file, a web page you crawl) can plant false facts or hidden instructions that are later retrieved for other users. In **fine-tuning**, bad examples can embed wrong behaviour or backdoors triggered by a special phrase. In **agent memory**, injected "facts" can persist across sessions. Defences: control **who can add content** and require review for sensitive sources; keep **provenance** (source, author, date) on every chunk and show citations; **sanitise** content on ingestion (strip hidden text and scripts); **version and scan** training data; test the finished system with questions whose answers you know; and let users **inspect and delete** stored memory. Treat your knowledge base as part of your attack surface.

## A poisoning-resistant ingestion checklist

Controls at the point where content enters the knowledge base.

```text
[ ] who can add or edit sources? least privilege + review for sensitive collections
[ ] provenance stored per chunk: source URL/file, author, ingestion date, content hash
[ ] hidden text, scripts and metadata stripped before indexing
[ ] untrusted web content kept in a separate, lower-trust index; answers cite sources
[ ] fine-tuning data versioned, scanned, and spot-checked for odd trigger phrases
[ ] canary questions with known answers run after every index update
[ ] users can inspect and delete stored memory; entries have an owner and an expiry
```

## Run canary questions after updates

Questions with known answers reveal a poisoned or broken index quickly.

**Quiz:** Why store provenance (source, author, date) with each chunk?

- [ ] To make chunks larger
- [x] To trace, cite and remove poisoned or outdated content
- [ ] Because models require it
- [ ] There is no benefit

*Answer:* To trace, cite and remove poisoned or outdated content. You can only clean up what you can trace.