Databricks says scaling an agent app starts with checkpoints—not bigger compute

Databricks has published a useful corrective to the usual “add more compute” answer for slow AI applications: first prove where the request is waiting, make every session recoverable, and count the database connections that horizontal scaling will create.

Treat agent state as disposable until it is checkpointed

The guide divides application state into soft state, such as in-memory conversation context, tool outputs and plans, and hard state that must survive a restart. For an agent loop, Databricks recommends saving a checkpoint after each meaningful step and making steps idempotent so a retry does not repeat an irreversible action. Hard state belongs in Lakebase or Unity Catalog tables, allowing another instance to reconstruct the session if affinity changes.

That distinction matters because Databricks Apps offers only best-effort session affinity. Restarts, failures and rescaling can move a user to another instance. A routing key may improve affinity for stateful API clients, but the guide explicitly says it must not be treated as proof of identity.

Caching is also an authorization decision

The sharpest warning concerns shared caches. Databricks says an application-level cache does not automatically enforce Unity Catalog permissions. A cache key therefore needs the user’s verified identity plus query parameters—and sometimes the session—not an identity value supplied by the client. Data should be shared across users only when it is genuinely identical for everyone.

This is easy to miss in an analytics agent: caching a generated result can lower warehouse load while quietly bypassing the permissions that protected the original query. The safe default in the guide is per-user caching.

Scale the whole request path

Databricks recommends percentile targets such as p95 latency, separate cold-start measurements and time-to-first-token for streaming responses. It also advises instrumenting SQL, model and API calls with OpenTelemetry spans before production. If latency rises while application CPU stays low, the likely bottleneck is a blocking call, a query, a connection pool or another downstream service—not insufficient app compute.

The connection arithmetic is concrete. With one pool per worker, the upper bound is instances × workers per instance × maximum pool size. Five instances, eight workers each and ten pooled connections can therefore open as many as 400 connections before other clients or deployment overlap are counted. Per-user Lakebase access has another boundary: the guide says a branch allows 500 Postgres roles.

Test failure, not just throughput

The operating contract includes a 15-second shutdown window between SIGTERM and SIGKILL. Final cleanup and checkpoints must fit inside it, but important work should already have been persisted during normal operation. Databricks recommends load tests that stop and restart the app, then verify that sessions continue and no work is lost; for agent apps, it suggests testing once with a mocked model to isolate application throughput and again with the real model for end-to-end latency.

This is vendor guidance, not an independent benchmark, and it does not cover scaling every downstream Databricks service. Its value is the checklist: before buying a larger instance, prove that state is portable, caches preserve identity boundaries, shutdown is survivable and the database can absorb the multiplied connection count.