Comparing Engines: A Practical Checklist

Questions beat feature tables

Feature lists on vendor sites are similar and change monthly, so compare with questions about your workload. Scale: how many vectors and what dimension now and in two years; does it fit in memory; what does it do beyond RAM (disk indexes, quantisation)? Filtering: are filters evaluated during search or after; are payload indexes supported; how does it behave with a 1% filter? Search types: dense only, or sparse/keyword and hybrid with fusion; multi-vector; reranking hooks? Updates: how quickly does a write become searchable; how are deletes and re-indexing handled? Consistency and durability: transactions, write-ahead log, snapshots, replication? Operations: deployment (Kubernetes, managed), backups, monitoring, upgrades. Security: authentication, role-based access, encryption, tenant isolation, audit. Ecosystem and cost: client libraries, framework integrations, licence, support, total price. Score each candidate against your answers.

A scoring sheet

Fill it in with measured results from your own tests, not from marketing pages.

Criterion (weight)            Candidate A     Candidate B     Candidate C   how measured
recall@10 at p95 < 50 ms (5)   __/5            __/5            __/5          your golden set + load test
filter at 1% selectivity (4)   __/5            __/5            __/5          returns 10 rows? latency?
write-to-searchable delay (3)  __/5            __/5            __/5          insert, poll until found
ops effort (3)                 __/5            __/5            __/5          deploy, backup, upgrade drill
security features (4)          __/5            __/5            __/5          RBAC, encryption, tenant tests
cost at 2-year scale (4)       __/5            __/5            __/5          price sheet + memory estimate

Quick check: Why compare engines with questions about your workload rather than feature tables?

• Feature lists look alike and change often; behaviour on your data decides
• Features never matter
• Vendors forbid comparison
• Tables are always wrong