Zepto’s agent-evaluation loop offers a practical playbook for production NL2SQL

Zepto’s new account of running customer-support agents is not a text-to-SQL case study. It is still unusually useful to teams putting generated SQL into production because it makes the assurance loop concrete: instrument every step, turn failures into regression cases, and spend evaluation budget where mistakes matter most.

The scale changes the math

Zepto says its multi-agent system processes more than 100,000 support tickets a day. At that volume, the company notes, a 1% error rate produces thousands of bad outcomes. The system now fully manages more than 80% of tickets with human oversight; Zepto reports a 65% reduction in support cost, a 20% improvement in customer satisfaction and payback in under one month. Those are vendor-and-customer-reported results, not an independent benchmark, but the operating detail behind them is the more transferable part of the post. Databricks and Zepto

Two loops, one promotion gate

The architecture separates development evaluation from production monitoring. Every invocation emits traces containing prompts, retrieved documents, tool calls, latency and decision paths. A candidate version is compared with the production baseline on a golden dataset, and it is promoted only if it clears agreed quality, cost and performance gates. Production failures then feed the next development cycle. Databricks and Zepto

That maps cleanly onto NL2SQL. A trace should preserve the user question, selected schema context, generated SQL, execution result, retries and any final explanation. Promotion should depend on more than execution success: a query can run and still use the wrong grain, join path or time window. The practical lesson is to keep deterministic checks—syntax, permissions, row limits and result invariants—beside model-based judgments rather than replacing them.

MLflow’s documentation supports that split. It describes evaluation datasets as versioned Unity Catalog assets that can combine labeled examples, production traces and synthetic cases. Its scorer framework offers built-in judges, custom judges and deterministic code-based scorers, with the same scorer usable in development and production monitoring. Evaluation datasets Scorers and judges

Sample by risk, not uniformly

Zepto does not evaluate every live interaction. It says risk-weighted sampling evaluates roughly 18–20% of traffic—about 14,400 traces a day—while capturing 45–60% of edge cases. Sampling rises for high-value customers, recently changed flows, negative sentiment and fraud-prone inputs; the company reports issue detection within four to six minutes and an 86% lower review cost per issue than uniform sampling. Databricks and Zepto

For an NL2SQL service, equivalent risk signals include write-capable tools, sensitive tables, unfamiliar join paths, large scans, newly changed semantic models and low-confidence schema retrieval. Uniformly sampling 10% of queries would spend scarce review capacity on routine reads while missing exactly the queries most likely to be expensive or dangerous.

A deployment checklist

1. Trace the whole query path, not only the final SQL.
2. Maintain a versioned golden set that includes production failures and business-critical questions.
3. Gate releases against the current production baseline on accuracy, cost and latency.
4. Use deterministic validators for hard rules and calibrated judges only for criteria that require interpretation.
5. Weight production sampling by blast radius, then alert on regressions quickly.

Zepto’s strongest lesson is organizational: evaluation is not a final test suite. It is the control plane that decides what may reach production and what must return to development.