Keyword Search with TF-IDF

Rare words matter more

TF-IDF weights a word by how often it appears in a document (term frequency) and how rare it is across all documents (inverse document frequency), so "leave" counts for more than "the". Documents and queries become sparse vectors and are compared with cosine similarity. Keyword methods are fast, need no model, and are excellent for exact terms: product codes, error numbers, names, acronyms. Their weakness is the vocabulary mismatch: a question using different words from the document finds nothing.

TF-IDF over five policy snippets, run

I ran this plain-Python (standard library only) example. "carry over" and "hotel limit" questions find the right snippet. But "can I work from home" ranks the expense policy first and the remote-work policy second, because the document says "remote" and not "home": a vocabulary mismatch that embeddings would handle better.

DOCS = {
 "leave": "Employees get 24 days of paid leave per year. Unused leave up to 5 days can be carried over to the next year.",
 "remote": "Remote work is allowed up to 3 days per week with manager approval. Core hours are 11:00 to 16:00.",
 "expense": "Expenses above 5000 rupees need approval from a director. Submit receipts within 30 days.",
 "security": "Use a password manager and enable two factor authentication. Report lost laptops within 24 hours.",
 "travel": "Flights must be booked at least 14 days in advance. Hotel cost is capped at 6000 rupees per night.",
}
import math, re
from collections import Counter

def tok(t): return re.findall(r"[a-z0-9]+", t.lower())

names = list(DOCS)
docs = [tok(DOCS[n]) for n in names]
N = len(docs)
df = Counter(w for d in docs for w in set(d))
idf = {w: math.log(N / df[w]) + 1 for w in df}

def vec(words):
    tf = Counter(words)
    return {w: tf[w] * idf.get(w, 0) for w in tf}

def cos(a, b):
    dot = sum(a[w] * b.get(w, 0) for w in a)
    na = math.sqrt(sum(v * v for v in a.values())); nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

vecs = [vec(d) for d in docs]
def search(q, k=3):
    qv = vec(tok(q))
    return sorted(((round(cos(qv, v), 3), n) for n, v in zip(names, vecs)), reverse=True)[:k]

print(search("how many days of leave can I carry over"))
print(search("what is the hotel limit"))
print(search("can I work from home"))

Output:

[(0.541, 'leave'), (0.032, 'expense'), (0.025, 'travel')]
[(0.227, 'travel'), (0.128, 'leave'), (0.077, 'remote')]
[(0.171, 'expense'), (0.131, 'remote'), (0.118, 'leave')]

Normalise text the same way twice

Lowercase and tokenise documents and queries with the same function, otherwise matches silently disappear.

Quick check: When does keyword search fail most clearly?

• When the index is small
• When searching for an exact error code
• When the question uses different words from the document
• When the text is English