EMG RAG’s production snapshot measures the failure modes most NL2SQL demos omit

A newly published repository offers a unusually inspectable case study of natural-language analytics in a small production setting. EMG RAG says it is used by one countertop fabricator’s office to answer questions over sanitized Moraware job-tracking and QuickBooks data. Rather than present a single headline accuracy score, the project exposes separate measurements for routing, retrieval, answer correctness, latency and evaluator instability.

Four paths, measured separately

The system routes each question to structured SQL, semantic retrieval, a hybrid path, or refusal. Its published golden set contains 82 questions: 53 structured, 23 semantic, three hybrid and three refusal cases. Fifty-five are marked verified and 27 remain draft. Two full runs report 79-of-82 routing accuracy and 75-of-82 answers judged correct.

That separation matters. A wrong answer can originate in route selection, SQL generation, document retrieval or synthesis, and a single aggregate score hides where repair work belongs. The hybrid path is especially concrete: SQL first identifies relevant job IDs, then constrains note retrieval to those jobs instead of searching every note globally.

The repository also reports retrieval stages independently. Dense-only recall@10 is 0.280, reciprocal-rank fusion reaches 0.500, and reranking reaches 0.818 with MRR of 0.660. Reported faithfulness differs across the two full runs—0.962 and 0.924—even though the answering components run at temperature zero.

It audits the judge, too

EMG RAG attributes some remaining movement to its Sonnet evaluator. Nine judge passes over one saved answer set produced between 74 and 76 correct answers; one row split five positive judgments to four negative. The project therefore states a roughly one-answer measurement floor instead of treating one model-judge pass as ground truth.

The SQL lane is layered rather than prompt-only. It accepts one parsed SELECT, checks each column against the view named by its alias, forces a 200-row limit, uses a read-only database role and applies a 10-second statement timeout. The per-alias check was added after a query using a valid column on the wrong alias passed an earlier global whitelist.

Useful evidence, bounded scope

This is not a general NL2SQL benchmark. It covers one company, one data snapshot and office-scale concurrency; two questions may run at once. The underlying production data and retrieved-note evaluation files are private, while the public snapshot redacts or replaces identifying values. More than a third of the golden questions are still marked draft, and the repository currently has no declared software license.

Those limits should block broad leaderboard claims, but not erase the engineering lesson. The strongest part of EMG RAG is not its 75-of-82 result. It is the decision to publish which subsystem failed, how much the judge moves, what the safety validator missed, and where the dataset stops supporting inference.