What analytics-agent teams can borrow from Shopify’s partitioned harness

Shopify’s latest agent-harness write-up is about application security, not natural-language analytics. That distinction matters. But the operating pattern is unusually transferable: keep the model interchangeable, constrain what each run sees, prove outputs with an independent oracle, and leave credentials and state transitions to deterministic code.

Partition context instead of filling the window

Shopify says its Dispatch orchestrator catalogs a target repository and groups related files into coherent partitions. The team aims to fill roughly 20–30% of the active model’s context window with the initial partition, leaving room for exploration and tool use. Shared dependencies are included where needed, while architecture documents and API descriptions are persisted as reusable artifacts rather than rediscovered on every run.

For analytics-agent teams, the useful question is not whether code files equal database tables—they do not. It is whether an agent is being handed an entire catalog simply because the context window can hold it. Shopify’s method suggests a more testable alternative: define bounded domains, include the shared business objects they depend on, and reserve context capacity for the agent’s actual investigation.

Separate generation from proof

Dispatch runs hunting agents in parallel, then invokes verifier agents sequentially. The verifier uses a different model and must author and execute tests against each candidate finding. Shopify calls tests the oracle; sequential execution also avoids collisions in ports, databases and fixtures.

That separation is more important than asking one agent to critique itself. In an analytics deployment, the equivalent design question is whether SQL generation and answer acceptance are distinct stages. A candidate query can be checked against executable assertions, permissions, expected grain or curated cases before its result is presented as trusted. That is an adaptation of Shopify’s pattern, not a claim that Shopify uses it for NL2SQL.

Persist the expensive understanding

The first Dispatch run builds partitions and documentation. Later runs compare commits and scan only the diff, updating stored artifacts when the code changes. Shopify reports full scans costing about $50–$300 with public frontier models, versus roughly $5–$50 for incremental scans, depending on model and application size.

A data-agent team should read those numbers as evidence for the architecture, not as a budget estimate for analytics. The transferable idea is to make schema and policy changes explicit inputs to reevaluation rather than rebuilding understanding from scratch on every question.

Keep the control plane deterministic

Shopify recommends deterministic scripts where structured inputs or outputs matter, and says credentials, Git and storage should remain in deterministic code rather than be owned by agents. Its production system also deduplicates and scores findings in a Ruby post-processing stage before reporting them.

The practical deployment test is simple: if changing the underlying model forces a rewrite of identity handling, state persistence, validation or audit behavior, the model has leaked into the control plane. Shopify’s five-month evaluation of newer models found better discovery but also more candidates requiring confirmation. Its conclusion is the durable one: the harness—not the newest model—has to absorb that variability.