dlt-hub / dlt-hub/dlt

Add support for sdc2 strategy in source pg_replication

Open
#1,607 0 comments 1 reaction 0 assignees View on GitHub
community
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

Hello,

I would like to be able to keep track of all historical changes in the destination, ie add support scd2 for `pg_replication` source.

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

`pg_replication` replicates the current state of the postgresql source DB, but we would like to keep track of all historical changes.
When setting `write_disposition={"disposition": "merge", "strategy": "scd2"}`, strategy is overriden and set to `delete-insert`.

See [slack thread](https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1721220411348669):

> the pg_replication source emits write_disposition hints with each "data item" (here's the [code](https://github.com/dlt-hub/verified-sources/blob/cb96e6923ac70be236cd818017348d303e045ff4/sources/pg_replication/helpers.py#L721) that determines which disposition is used for a certain item).
Setting the write disposition at the pipeline or resource level won't help. Those hints are overridden. The scd2 merge strategy did not yet exist when pg_replication was built.
It would be good if we add this info to the pg_replication docs.

_Note_: `scd2` in source `sql_database` only adds diffs between active run and last run, we would like to get events even if a row was deleted then recreated for instance. Therefore it is better to use Change Data Capture in our case, and keep all data item history.

### Proposed solution

Rewrite [pg_replication helper](https://github.com/dlt-hub/verified-sources/blob/cb96e6923ac70be236cd818017348d303e045ff4/sources/pg_replication/helpers.py) ?

### Related issues

Documentation for `pg_replication` does not specify that `scd2` is not supported

Contributor guide

Open the contributing guide

Research direction

Start with sources/pg_replication/helpers.py, especially the write-disposition logic linked in the issue, and review the pg_replication documentation. Clarify how scd2 should preserve deletion and recreation history and how per-item hints interact with merge settings. Done means the requested strategy is supported and the documentation states its behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.