The agentic data stack is deleting pipelines—not correctness contracts
AWS, Databricks and Google are removing migration and transport layers. Each launch exposes a different obligation that still belongs to the operator.
Three first-party launches point in the same direction: cloud data platforms are removing infrastructure that used to sit between operational data and analytical or agent workloads. AWS now manages Hive-to-Iceberg migration, Databricks can take Debezium change events without Kafka as transport, and Google’s AlloyDB can read Iceberg tables in place. The important pattern is what those products do not remove.
Fewer pipelines means sharper boundaries
AWS S3 Tables can copy Glue-registered Hive tables stored as Parquet into a new managed Iceberg table. It compacts files, converts compatible types and supports incremental synchronization while the source stays available. But AWS also produces compatibility reports and lets teams pause before copying. The service automates movement; the operator still decides whether schema conversions are acceptable and when the destination is ready for cutover.
Databricks removes a different layer. Its Debezium Server sink sends database changes directly to Zerobus Ingest and Unity Catalog Delta tables, avoiding Kafka when Kafka would exist only as transport. Yet the delivery contract is explicitly at least once. Databricks says an earlier durable group can be delivered again after a later failure, so downstream processing must deduplicate with the event’s idempotency key and order changes by source position. Removing the broker does not create exactly-once business state.
Google’s AlloyDB preview removes another pipeline by reading Iceberg tables directly from Cloud Storage and joining them with live operational tables in one PostgreSQL query. Google also offers two alternatives with different freshness and execution contracts: scheduled AlloyDB sync creates a local copy whose freshness is bounded by its last refresh, while federation leaves processing in BigQuery and returns results to AlloyDB. “No custom pipeline” therefore does not mean “no architecture choice.” Teams still choose between in-place reads, copied local state and remote execution.
The control plane becomes the product
Across all three launches, the disappearing component is visible infrastructure: migration compute, Kafka transport or a custom synchronization path. The surviving obligations are less visible but more consequential:
- AWS: approve schema policy and cutover after reviewing migration evidence.
- Databricks: deduplicate redeliveries and reconstruct source order before presenting current state.
- Google: define the freshness, latency and billing boundary for each access mode.
That distinction matters for data agents. An agent can query a table exposed through any of these paths without knowing whether the table is awaiting cutover, contains duplicate change events or reflects last night’s scheduled refresh. Connectivity is not a correctness signal.
The practical response is to publish those contracts beside the data: migration status and validation results for newly converted tables; source positions and deduplication rules for CDC feeds; and freshness plus execution-location metadata for federated or synchronized data. The platforms are making the plumbing smaller. Teams should use the saved operational effort to make these remaining boundaries more explicit—not assume the boundaries disappeared with the pipeline.
sources
- AWS: Migrating tables to S3 Tablesdocs.aws.amazon.com
- Databricks: Push-based CDC into Unity Catalog with Zerobus Ingest and Debeziumwww.databricks.com
- Google Cloud: Direct Iceberg access in AlloyDBcloud.google.com
comments · 0