“Live data” hides three different freshness contracts for analytics agents
Databricks, Confluent and Google all promise current data without traditional batch copies—but they differ on checkpoints, maintained state, replay and who owns correctness.
Enterprise vendors increasingly describe agent-accessible data as “live,” “current” or queried “in place.” Those phrases sound interchangeable, but recent first-party material from Databricks, Confluent and Google describes three materially different operating contracts.
The distinction matters because an analytics agent can return a recent-looking answer while still relying on different guarantees about updates, deletes, replay and authorization. Before treating “live” as an architecture, operators need to ask what is actually moving, where state is maintained, and which system is authoritative.
Contract 1: incremental consumption with a checkpoint
Databricks’ OpenSharing implementation guide exposes a shared Delta table and its history to a recipient. A Databricks recipient can use Structured Streaming and a durable checkpoint to process only versions it has not consumed. If it needs a local analytical copy, it materializes that data downstream.
But the change semantics are a deployment choice. Databricks says its ignoreChanges pattern may re-emit rows and does not propagate deletes. Consumers that require inserts, updates and deletes must use Change Data Feed. Non-Spark recipients do not get the same continuous-stream path; they pull changes by version range. Here, “live” means an ordered history is available, while the recipient still owns its checkpoint, target and reconciliation policy.
Contract 2: continuously maintained shared state
Confluent’s Streamhouse proposal puts more responsibility into the middle layer. Change capture and a durable event backbone feed stateful processing, which continuously maintains reusable representations for applications, agents and analytics. Consumers can receive changes, query current state or read materialized tables.
Confluent is explicit that this does not create a perfectly synchronized global “now.” Different implementations still trade latency, consistency, replay, cost and operational complexity. Operational systems remain responsible for authorizing and committing transactions. The contract is therefore not merely access to source history; it is maintained state plus replay and shared context, with source systems retaining transactional authority.
Contract 3: federated query under the user’s identity
Google’s Gemini Enterprise Data Cloud connectors use federated-query mode to query BigQuery, Spanner, Cloud SQL, AlloyDB and Looker without copying or indexing the data into a Gemini data store. Gemini discovers accessible metadata through Knowledge Catalog, generates SQL and executes it under the end user’s credentials, preserving existing row- and column-level controls.
This path avoids maintaining a separate analytical replica inside Gemini. It also shifts the contract toward source availability, query-time permissions and generated-query correctness. Google separates broad exploratory connectors from curated Conversational Analytics agents, which data teams scope to selected data and certified use cases.
Ask for the contract, not the adjective
These designs can coexist. A company might maintain current operational state through streams, publish governed history to partners and let users query selected sources in place. The mistake is assuming that all three uses of “live” promise the same thing.
For every agent-facing source, document four facts: whether reads are federated or materialized; how progress and replay are tracked; how updates and deletes propagate; and which identity authorizes the final read or action. “Live” is a marketing adjective. Those four answers are the production contract.
sources
- Read, don't copy: replacing file feeds with OpenSharing — Part 2www.databricks.com
- Your Business, Always Current: Streamhousewww.confluent.io
- Query enterprise data in Gemini with Data Cloud connectorscloud.google.com
comments · 0