Snowflake can now shadow production traffic—but comparison still needs a scorecard

Snowflake’s September 8 preview adds a useful missing primitive for production evaluation: a gateway can send every request to a primary service while mirroring a chosen percentage to one or more shadow endpoints. The primary still produces the user-facing response; outputs from shadow endpoints are not returned to clients. That creates a safer path for testing a new model, prompt, retrieval stack or inference service against real request patterns without putting the challenger directly in the serving path.

The distinction from ordinary A/B routing matters. Snowflake’s gateway documentation says percentage-based traffic splitting sends different requests to different serving endpoints. Shadow mode instead gives the primary and challenger copies of the same incoming requests. For comparative evaluation, that paired design removes one source of noise: differences in request mix.

But shadowing is transport, not a benchmark. Snowflake documents routing behavior, not an automatic verdict on which endpoint is better. Teams still need to capture paired outputs and score the dimensions that matter for their workload.

Build the scorecard before enabling the mirror

For an analytics or NL2SQL service, the minimum comparison should include:

• Answer or execution correctness: run both outputs against the same accepted result or task-specific oracle.
• Policy behavior: verify that the challenger refuses or constrains the same unauthorized requests as the primary.
• Latency and failures: compare tail latency, timeout rate and malformed-response rate, not only averages.
• Cost: normalize token, compute and warehouse consumption per completed request.
• Coverage: report how many mirrored requests produced a scoreable pair; dropped or unparseable challenger outputs must not disappear from the denominator.

Stratify those results by workload class—simple lookup, aggregation, multi-table join, ambiguous request and unsupported request—before promoting anything. An overall win can conceal a regression in the small slice of requests that carries the highest business or security risk.

Operational details also affect the experiment. Snowflake says a gateway can target five endpoints by default, configuration changes take effect within a minute, and routing uses only healthy endpoints. Its readiness state may take up to 40 seconds to refresh. Evaluators should therefore record gateway configuration, endpoint health and the exact observation window alongside every score; otherwise a partial mirror or unhealthy challenger can look like missing evaluation data.

There are access boundaries too. Gateway users receive access through USAGE on the gateway rather than service roles on the underlying endpoints, and gateways currently do not support caller’s rights, CORS or CSP headers. Those differences mean a shadow test should validate authorization behavior through the gateway itself, not assume it matches direct endpoint access.

The preview is valuable because it turns production traffic into paired evaluation input without exposing users to challenger responses. The promotion decision, however, remains a benchmark-design problem: fixed metrics, complete denominators, risk-weighted slices and a reproducible observation window.