debezium / debezium/dbz

Stream ongoing changes from the CDC log with a clean snapshot handoff

Open
#2,160 1 comment 0 reactions 1 assignee Claimed by @siddhantcvdi View on GitHub
component/sqlite-connector type/task
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Stream ongoing changes from `_debezium_cdc_log` and hand off from the snapshot with no gaps and no
duplicates. This is the SQLite connector's first live change data capture.

This is part of the SQLite connector tracked in issue #2056. It builds on the config, schema, and
snapshot work in #2069.

## What this adds

The connector already boots, loads the schema, and takes an initial snapshot that records a high-water
mark. This issue makes it stream. After the snapshot, the connector polls `_debezium_cdc_log` for new
rows and emits each one as a standard Debezium change event:

- A poll loop reads new rows in `change_id` order, in small bounded batches, and sleeps between polls
when there is nothing new. Each query opens and closes a short read transaction so it does not block
SQLite from checkpointing the WAL.
- Each row becomes a `c`, `u`, or `d` change event. The before and after row data stored as JSON in the
log is decoded into the column structure the connector already knows from the schema, and the framework
builds the key, value, and envelope.
- The offset advances after every row, so a stop part way through a batch re-reads only the rows after
the last one emitted.
- The `source` block of each event carries the change position and the commit time: `ts_ms` comes from
the row's `committed_at`, and the source metadata exposes the `change_id` and the table name.

## The handoff

The main correctness goal is a clean handoff from snapshot to streaming. While the snapshot reads the
tables, the generator keeps writing new rows to the log. The snapshot records the largest `change_id`
visible inside its read view as a high-water mark, and streaming resumes at the first `change_id` past
it. Every change committed after the snapshot's view opened has a larger id, so it is absent from the
snapshot data and delivered once by streaming. No change is missed and none is delivered twice.

## Scope

- This issue covers the poll loop, the change event emitter, and the handoff from the snapshot to
streaming.
- The work is tested against the SQL trigger stand-in, which writes the same `_debezium_cdc_log` shape
the native extension will write, so the streaming code does not depend on which generator is in use.
The integration tests run against a temporary database file with no Docker.

## Done when

- A live insert, update, and delete each arrive as the matching change event with the right before and
after values and source metadata.
- A write committed during the snapshot window is delivered by streaming exactly once.
- `./mvnw verify` is green, unit and integration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.