BM25: The Standard Keyword Ranker

TF-IDF with saturation and length control

BM25 improves on TF-IDF with two ideas: term-frequency saturation (the tenth occurrence of a word adds much less than the second) and length normalisation (a match in a short document counts more than the same match in a very long one). Two parameters, usually k1 around 1.2 to 2 and b around 0.75, control these. BM25 is the default in Elasticsearch, OpenSearch and Lucene and is a strong baseline that often beats poorly chosen embeddings on exact-term queries. Always keep it available, even if you add vectors.

BM25 from scratch, run

I ran this plain-Python (standard library only) example. "expense approval director" ranks the expense policy first (2.534) and "lost laptop report" ranks the security policy first (3.024); the other documents score near zero.

DOCS = {
 "leave": "Employees get 24 days of paid leave per year. Unused leave up to 5 days can be carried over to the next year.",
 "remote": "Remote work is allowed up to 3 days per week with manager approval. Core hours are 11:00 to 16:00.",
 "expense": "Expenses above 5000 rupees need approval from a director. Submit receipts within 30 days.",
 "security": "Use a password manager and enable two factor authentication. Report lost laptops within 24 hours.",
 "travel": "Flights must be booked at least 14 days in advance. Hotel cost is capped at 6000 rupees per night.",
}
import math, re
from collections import Counter

def tok(t): return re.findall(r"[a-z0-9]+", t.lower())
names = list(DOCS); docs = [tok(DOCS[n]) for n in names]
N = len(docs); avg = sum(len(d) for d in docs) / N
df = Counter(w for d in docs for w in set(d))

def bm25(q, d, k1=1.5, b=0.75):
    tf = Counter(d); score = 0.0
    for w in tok(q):
        if w not in tf: continue
        idf = math.log((N - df[w] + 0.5) / (df[w] + 0.5) + 1)
        score += idf * tf[w] * (k1 + 1) / (tf[w] + k1 * (1 - b + b * len(d) / avg))
    return score

for q in ("expense approval director", "lost laptop report"):
    ranked = sorted(((round(bm25(q, d), 3), n) for n, d in zip(names, docs)), reverse=True)[:3]
    print(q, "->", ranked)

Output:

expense approval director -> [(2.534, 'expense'), (0.823, 'remote'), (0.0, 'travel')]
lost laptop report -> [(3.024, 'security'), (0.0, 'travel'), (0.0, 'remote')]

Keep BM25 for identifiers

Order numbers, error codes and names are often missed by embeddings. A keyword index catches them reliably.

Quick check: What does BM25 length normalisation do?

• Prevents long documents from winning just by containing more words
• Shortens the query
• Encrypts text
• Chooses the embedding model