debezium / debezium/dbz

Demo: Keep *all* change events in a downstream database [DBZ-2101]

Open
#313 5 comments 0 reactions 1 assignee Claimed by @SaidiBTW View on GitHub
component/examples good first issue migrated-from-jira type/task
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2101](https://issues.redhat.com/browse/DBZ-2101)

A while ago, we showed on the blog how to [create an audit log|https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing/] based on change data events.

That example didn't cover though how those enriched change events are propagated to a sink database or DWH, allowing to analyse the full change history of a given customer or purchase order for instance. In this demo,

* take those enriched change events and the [flattening SMT|https://debezium.io/documentation/reference/configuration/event-flattening.html], to extract only the current row state (after)
* that SMT allows to add additional attributes from the record to the emitted flat row, so we could propagate that user name etc.
* always insert the records on the sink side instead of doing upserts

The Apache Camel Kafka Connect JDBC sink connector should be used on the sink side. Or perhaps the sink connectors for Google Big Query, Snowflake or another DWH. The sink primary key would be derived from the actual table PK + uniquely identifying attributes from the source block of change events (which would have to be propagated to the emitted row using the flattening SMT). I.e. each event would have its own unique PK value. While each event only ever *should* be inserted once into the sink database, this could happen another time after a connector crash ("at least once" semantics"), so upsert semantics should be applied to effectively ignore these duplicates.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.