Conversion cut 70,000 prompt tokens by moving schemas behind validators

A large context window can hide an architectural tax. Conversion says the median request to its production marketing agent carried about 150,000 tokens, mixing instructions, tool schemas and returned data into every turn. Its remedy was not a larger model window. It moved knowledge out of the prompt and gave the agent ways to retrieve or validate it only when needed. Conversion’s engineering account is unusually concrete about the resulting trade-offs.

Move instructions behind an explicit lookup

Conversion moved roughly 37,000 tokens of format guidance into read-only “skill” files mounted in the agent sandbox. Tool descriptions point the agent to the relevant file, so an email task loads the email instructions while an unrelated metrics question does not. The important design detail is discoverability: removing instructions without leaving a pointer made the agent faster but less capable, so every removal needed a replacement path to the same knowledge. Source

That is directly applicable to analytics agents. SQL dialect notes, metric definitions and workflow examples do not all need to ride in every prompt, but the agent must know that they exist and when to retrieve them.

Keep the validator, not the recursive schema

Four recursive email and form tool schemas contributed about 70,000 tokens to every request, Conversion reports. The team replaced those model-visible schemas with plain JSON-object inputs, documented the construction rules in skills and retained the full schemas inside deterministic validators. Invalid payloads return an exact error and a pointer to the relevant skill, letting the model repair its draft. Source

Conversion measured the latency cost across one month of Claude Opus 5 production traffic. Below 200,000 prompt tokens, it found time to first token rose by roughly 31 milliseconds per additional 1,000 tokens; it estimates the removed 70,000 schema tokens had added about two seconds to every turn. The company also warns that caching changes the curve, so teams should measure their own provider and traffic pattern rather than treat that rate as universal. Source

Put large query results in files

Tool output was the other pressure point. Conversion writes any result over 20,000 characters to a sandbox file and returns only its path plus a 1,000-character preview. The agent can grep for the relevant rows or process the file with code instead of repeatedly emitting data through inference. Source

For NL2SQL systems, this suggests a practical boundary: generated SQL may belong in the conversational trace, but a thousand-row result set probably does not. Persist the result, preview it, and expose deterministic operations for filtering, aggregation and validation.

The broader lesson is not “hide context.” It is to compile repeated knowledge into retrievable artifacts and deterministic checks. Conversion still identifies an unresolved risk: with more than 100 first-party tools, tool search can reduce prompt size while making undiscovered capabilities silently invisible. Context reduction therefore needs regression benchmarks that test capability discovery—not just token counts and latency.