A dataset-search benchmark gains six recall points by turning metadata requests into SQL

The result

A new dataset-search evaluation reports that combining semantic retrieval with SQL-based metadata filtering raises Recall@10 from 0.644 to 0.704 on the held-out KaggleDS test set—a gain of 6.0 percentage points. The system first uses Hypothetical Schema Embeddings, or HySE, to retrieve 50 candidate tables, then applies FlexPrune once to translate natural-language metadata constraints into SQL predicates and intersect them with those candidates.

That distinction matters. The result is not a text-to-SQL accuracy score and it is not measured on BIRD or Spider. It measures whether one designated target table appears among the top 10 results after a user describes an analytical task and metadata requirements.

What was tested

KaggleDS contains 6,503 tables from 3,141 Kaggle datasets. Its held-out test split contains 7,011 task queries over 2,337 tables; the split is made at dataset level so test schemas do not also occur in training. HySE uses gpt-4o-mini to generate two hypothetical schemas per query, embeds those schemas and the original request, and fuses the representations before retrieval.

On the same test split, HySE beat direct task-query embedding by 2.0 points at Recall@10 and beat a TAPAS Dense Table Retriever fine-tuned on the KaggleDS training split by 3.9 points at Recall@10. The paper also evaluates transfer on the NQ-Tables test split, comprising 959 questions against 169,898 tables, and reports gains over the supervised DTR-large baseline without NQ-Tables training.

Why the benchmark is useful—and bounded

The strongest signal is architectural: semantic expansion and symbolic filtering solve different parts of discovery. HySE broadens the candidate set around the user’s intent; FlexPrune narrows it with executable metadata constraints. The six-point chained gain is therefore more informative than treating one embedding model as the whole search system.

But KaggleDS assigns exactly one gold table per query. Real analysis requests may be satisfied by several datasets, so Recall@10 can mark useful alternatives as wrong. The corpus also comes from one relatively well-documented source, and the paper does not provide a scaling curve or test FlexPrune on sparse enterprise-catalog metadata. The NQ-Tables transfer result strengthens the retrieval claim, but cannot test FlexPrune because that corpus lacks the required metadata fields.

The practical verdict is narrower than “LLMs solve dataset discovery.” The evaluation shows that a staged retriever can improve a clearly defined held-out metric. Whether the same gain survives incomplete lineage, stale descriptions and multiple valid tables remains the reproduction test that matters next.