Neo4j’s benchmark exposes metadata discovery as Text2SQL’s hidden tax

Neo4j has published a pair of reproducible Text2SQL experiments that isolate a cost most evaluations blur into the final answer: how much work an agent does before it identifies the right tables. The vendor-authored tests report that retrieving schema context from a Neo4j/Neocarta metadata graph reduced token use by 45% to 81% on a BigQuery catalog and by 19% to 54% on a Databricks catalog. Both harnesses are available as source code, so teams can inspect—or rerun—the comparison rather than relying only on charts in a product post.

The experiment separates discovery from SQL generation

The setup keeps business data in the warehouse. Neocarta ingests metadata—databases, schemas, tables, columns, comments and declared relationships—into Neo4j. At question time, semantic search identifies an anchor table and graph traversal expands the relevant columns and join paths. The language model then writes SQL, which executes against BigQuery or Databricks.

That distinction matters. In the baseline, the agent lists schemas and tables, inspects columns and repeatedly searches the catalog. With the graph layer, it retrieves a compact context bundle before generating SQL. The benchmark is therefore testing a metadata-discovery architecture, not whether Neo4j can answer the business query itself.

The strongest result is reduced search work

On BigQuery’s public Census ACS catalog—278 similar tables and roughly 68,000 columns—the compact-retrieval test covered four verified questions across five models. The published results show 45% to 81% fewer tokens. GPT-4o-mini improved from two correct answers out of four to four out of four, while the other tested models remained at four out of four in both modes.

The Databricks harness uses a synthetic legacy-style lakehouse with 264 tables, 5,614 columns and 12 schemas. Across 11 questions and eight Anthropic models, the graph-assisted runs used 19% to 54% fewer tokens; all eight reached 11 out of 11 with the layer. In one captured question, the baseline made 18 warehouse round trips—17 catalog probes plus the business query—while the graph-assisted path sent only the business query to Databricks.

For practitioners, that is the useful finding: a data agent can spend a material share of its latency and token budget rediscovering metadata on every prompt, even when it ultimately generates the same SQL.

Treat the numbers as a local test, not a universal benchmark

The evidence has clear limits. Neo4j published the analysis, a Neo4j solutions engineer authored it, and Neocarta is the tested retrieval layer. The BigQuery accuracy sample is only four questions. The Databricks catalog is synthetic, its 11 questions use deterministic expected-table and answer checks, and the experiments do not compare the graph approach against every plausible retrieval baseline.

The right next step is not to accept “81%” as a procurement promise. It is to rerun the open harness against a representative catalog, preserve the same model and questions across both modes, and measure table selection, warehouse round trips, tokens, latency and answer correctness separately. Neo4j’s contribution is a testable claim: in large, opaque catalogs, schema discovery may be the first Text2SQL bottleneck worth removing.