SQLite’s semantic-search win is a reranking result, not a vector-database replacement

A new evaluation of scrydb, a Python search library built on SQLite FTS5 and sqlite-vec, makes a useful claim for analytics teams—but a narrower one than “SQLite replaces the vector database.” The paper evaluates lexical, semantic and hybrid retrieval across eight BEIR datasets using Qwen3-Embedding-8B. Retrieval ran without a GPU on an Apple M2 MacBook Air with 24 GB of memory; document and query embeddings were computed beforehand through a remote API. Source: paper and evaluation setup

The result that matters

The strongest efficiency result is binary-first reranking. A Hamming-distance scan followed by int8 cosine reranking matched exhaustive int8 search in P@10 and nDCG@10 to three decimals on all eight datasets, while averaging 164.5 ms per query instead of 822.5 ms. That is roughly one-fifth of the latency. The trade-off still appears deeper in the ranking: on TREC-COVID, average precision was 0.407 for Hamming-plus-int8 versus 0.413 for exhaustive int8. Source: effectiveness and latency results

Plain Hamming retrieval averaged 22.9 ms across the eight datasets, compared with 822.5 ms for exhaustive int8 cosine and 1,998.1 ms for exhaustive float32 cosine. On Quora, the largest evaluated corpus at 523,000 documents, Hamming search took 81.5 ms per query. The benchmark therefore supports a specific systems pattern: use compact binary codes to find a candidate set, then spend higher-precision compute only on that set. Source: latency analysis

Quality is close, not identical

Across the study, the best scrydb configuration for each dataset trailed the paper’s full-precision MTEB reference by a mean 0.006 nDCG@10 and met or exceeded it on four of eight datasets. One recorded example is FiQA: Hamming-plus-int8 reached 0.649 nDCG@10, versus 0.646 for the cited MTEB baseline. But the comparison is not a clean storage-engine bake-off. The authors’ own exhaustive float32 run scored 0.885 on TREC-COVID against the reported MTEB reference of 0.950, indicating that embedding or prompt-pipeline differences existed before quantization entered the picture. Source: effectiveness discussion

The boundary for NL2SQL stacks

This is a retrieval benchmark, not a text-to-SQL benchmark. It says nothing directly about execution accuracy, schema linking or generated SQL. Its relevance is upstream: schema documentation, business definitions and example-value retrieval can often fit the evaluated small-to-medium regime.

The paper is explicit about the limit. scrydb performs an exhaustive scan rather than approximate-nearest-neighbor search and does not target multi-tenant serving, sharding, replication or continuously growing web-scale corpora. The measured Hamming latency scales at about 0.15 ms per thousand documents, plus roughly 1.3 ms of fixed overhead; the authors place the practical regime at up to a few million documents, not tens of millions or billions. Source: benchmark boundary

For NL2SQL teams, the actionable takeaway is modest: benchmark binary-first SQLite retrieval before accepting the operational cost of a separate vector service. If the corpus or concurrency profile crosses that boundary, the paper itself says to move on.