# Benchmarking Honestly — Vector Databases

Source: https://www.geekswithgeeks.com/en/vector-databases/e-bench

> Measure recall, latency and throughput together, on your data.

## Speed without recall is meaningless

A fast ANN index can be fast simply because it returns worse results, so always report **recall together with latency**. A sound benchmark uses **your data** (or realistic data with the same dimension and clustering), **realistic queries** and **filters**, a fixed **recall target** (such as 0.95), and measures **p50, p95 and p99 latency**, **throughput at concurrency** (queries per second with 1, 8, 32 clients), **index build time**, **memory and disk**, **write throughput** and **freshness**, on **identical hardware** with warm caches and enough repetitions. Compare the **cheapest configuration that reaches the target recall** for each system, not default settings. Beware of vendor benchmarks that use easy data, tiny filters or tuned settings you will not get in production.

**Quiz:** Why must a benchmark report recall along with latency?

- [ ] Latency is meaningless
- [ ] Recall is a type of latency
- [ ] It is required by SQL
- [x] A system can look fast simply by returning worse results

*Answer:* A system can look fast simply by returning worse results. Compare engines at equal recall, not at default settings.