Benchmarking Honestly

Speed without recall is meaningless

A fast ANN index can be fast simply because it returns worse results, so always report recall together with latency. A sound benchmark uses your data (or realistic data with the same dimension and clustering), realistic queries and filters, a fixed recall target (such as 0.95), and measures p50, p95 and p99 latency, throughput at concurrency (queries per second with 1, 8, 32 clients), index build time, memory and disk, write throughput and freshness, on identical hardware with warm caches and enough repetitions. Compare the cheapest configuration that reaches the target recall for each system, not default settings. Beware of vendor benchmarks that use easy data, tiny filters or tuned settings you will not get in production.

Quick check: Why must a benchmark report recall along with latency?

• Latency is meaningless
• Recall is a type of latency
• It is required by SQL
• A system can look fast simply by returning worse results