Databricks removes Kafka from this CDC path—but not the duplicate-delivery problem

Databricks and NVIDIA have documented a direct path from a database transaction log into Unity Catalog: run Debezium Server beside the source, then push its change events to Databricks Zerobus Ingest. The design can remove Kafka and a separate Spark ingestion hop when the lakehouse is the only destination, and it needs only outbound connectivity from the source network. But simplifying transport does not make the pipeline exactly-once.

What actually shipped

The Zerobus sink is part of Debezium Server 3.7. Databricks built the typed sink; NVIDIA added an envelope mode that retains the operation, key, value, source position and an idempotency key in a fixed nine-column landing table. The sink waits for Zerobus to acknowledge a contiguous record group as durable before Debezium advances its source offset.

That is a useful failure boundary: an event is not acknowledged merely because it left the source process. It is not, however, an exactly-once guarantee. If a later group fails, a previously durable group can be delivered again. Databricks explicitly describes delivery as at-least-once and says duplicates must be handled downstream.

Treat Bronze as a changelog, not a replica

Envelope mode’s idempotency_key is a SHA-256 identity derived from destination, partition, key, operation and source position. A replay receives the same identity, which gives the Silver transformation a stable deduplication key. It does not remove the duplicate automatically, order events or turn the append-only Bronze table into current state.

A production pipeline therefore needs an explicit promotion contract:

1. Land the full envelope in Bronze and preserve the source position.
2. Deduplicate on idempotency_key before applying state changes.
3. Sequence updates by the database’s source position—for PostgreSQL, the write-ahead-log LSN—not by arrival time.
4. Interpret deletes as state transitions rather than discarding their null after image.
5. Build the current-state Type-1 or historical Type-2 table only after those controls pass.

Test the replay path before cutover

The most important acceptance test is a controlled failure between two acknowledged groups. Restart the source-side process, verify that a durable group may reappear, and prove that the promoted table remains unchanged after deduplication. Then test insert-update-delete ordering for the same primary key and an oversized record, which the sink rejects without committing its offset.

Teams should also keep Kafka when it serves purposes beyond transport: multiple independent consumers, long retention and replay, or an existing event-bus operating model. Databricks’ comparison is narrower: when Delta is the sole destination, the direct sink removes infrastructure that otherwise exists only to relay CDC.

The new route is operationally smaller, not responsibility-free. The source team still runs Debezium Server; the lakehouse team still owns replay-safe transformations, monitoring and reconciliation. The architecture earns its simplicity only when those responsibilities are written down and tested.