# Keyword Search with TF-IDF — Retrieval-Augmented Generation (RAG)

Source: https://www.geekswithgeeks.com/en/rag/v-tfidf

> Score documents by weighted word overlap and see where it fails.

## Rare words matter more

**TF-IDF** weights a word by how often it appears in a document (**term frequency**) and how rare it is across all documents (**inverse document frequency**), so "leave" counts for more than "the". Documents and queries become sparse vectors and are compared with cosine similarity. Keyword methods are fast, need no model, and are excellent for **exact terms**: product codes, error numbers, names, acronyms. Their weakness is the **vocabulary mismatch**: a question using different words from the document finds nothing.

## TF-IDF over five policy snippets, run

I ran this plain-Python (standard library only) example. "carry over" and "hotel limit" questions find the right snippet. But "can I work from home" ranks the expense policy first and the remote-work policy second, because the document says "remote" and not "home": a vocabulary mismatch that embeddings would handle better.

```python
DOCS = {
 "leave": "Employees get 24 days of paid leave per year. Unused leave up to 5 days can be carried over to the next year.",
 "remote": "Remote work is allowed up to 3 days per week with manager approval. Core hours are 11:00 to 16:00.",
 "expense": "Expenses above 5000 rupees need approval from a director. Submit receipts within 30 days.",
 "security": "Use a password manager and enable two factor authentication. Report lost laptops within 24 hours.",
 "travel": "Flights must be booked at least 14 days in advance. Hotel cost is capped at 6000 rupees per night.",
}
import math, re
from collections import Counter

def tok(t): return re.findall(r"[a-z0-9]+", t.lower())

names = list(DOCS)
docs = [tok(DOCS[n]) for n in names]
N = len(docs)
df = Counter(w for d in docs for w in set(d))
idf = {w: math.log(N / df[w]) + 1 for w in df}

def vec(words):
    tf = Counter(words)
    return {w: tf[w] * idf.get(w, 0) for w in tf}

def cos(a, b):
    dot = sum(a[w] * b.get(w, 0) for w in a)
    na = math.sqrt(sum(v * v for v in a.values())); nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

vecs = [vec(d) for d in docs]
def search(q, k=3):
    qv = vec(tok(q))
    return sorted(((round(cos(qv, v), 3), n) for n, v in zip(names, vecs)), reverse=True)[:k]

print(search("how many days of leave can I carry over"))
print(search("what is the hotel limit"))
print(search("can I work from home"))

```

Output:

```
[(0.541, 'leave'), (0.032, 'expense'), (0.025, 'travel')]
[(0.227, 'travel'), (0.128, 'leave'), (0.077, 'remote')]
[(0.171, 'expense'), (0.131, 'remote'), (0.118, 'leave')]
```

## Normalise text the same way twice

Lowercase and tokenise documents and queries with the same function, otherwise matches silently disappear.

**Quiz:** When does keyword search fail most clearly?

- [ ] When the index is small
- [ ] When searching for an exact error code
- [x] When the question uses different words from the document
- [ ] When the text is English

*Answer:* When the question uses different words from the document. Exact-word matching cannot connect different wording of the same idea.