Add support for sdc2 strategy in source pg_replication
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
Hello,
I would like to be able to keep track of all historical changes in the destination, ie add support scd2 for `pg_replication` source.
### Are you a dlt user?
Yes, I'm already a dlt user.
### Use case
`pg_replication` replicates the current state of the postgresql source DB, but we would like to keep track of all historical changes.
When setting `write_disposition={"disposition": "merge", "strategy": "scd2"}`, strategy is overriden and set to `delete-insert`.
See [slack thread](https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1721220411348669):
> the pg_replication source emits write_disposition hints with each "data item" (here's the [code](https://github.com/dlt-hub/verified-sources/blob/cb96e6923ac70be236cd818017348d303e045ff4/sources/pg_replication/helpers.py#L721) that determines which disposition is used for a certain item).
Setting the write disposition at the pipeline or resource level won't help. Those hints are overridden. The scd2 merge strategy did not yet exist when pg_replication was built.
It would be good if we add this info to the pg_replication docs.
_Note_: `scd2` in source `sql_database` only adds diffs between active run and last run, we would like to get events even if a row was deleted then recreated for instance. Therefore it is better to use Change Data Capture in our case, and keep all data item history.
### Proposed solution
Rewrite [pg_replication helper](https://github.com/dlt-hub/verified-sources/blob/cb96e6923ac70be236cd818017348d303e045ff4/sources/pg_replication/helpers.py) ?
### Related issues
Documentation for `pg_replication` does not specify that `scd2` is not supported
Contributor guide
Research direction
Start with sources/pg_replication/helpers.py, especially the write-disposition logic linked in the issue, and review the pg_replication documentation. Clarify how scd2 should preserve deletion and recreation history and how per-item hints interact with merge settings. Done means the requested strategy is supported and the documentation states its behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100