MeltanoLabs / MeltanoLabs/tap-postgres
feat: potentially add xmin replication?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
Airbyte has this as of July 10, see here. Haven't spent time scoping this out but doesnt work on current version of this plugin. I suspect this'd be easy to add, I assume the catalog is excluding hidden/system columns and it could just include them(?)
Specifically, I tried to just set xmin as the replication column, and it looks like it errors saying hey that column isnt in the catalog.
2023-08-25T17:06:34.041196Z [info ] time=2023-08-25 17:06:31 name=tap_postgres level=INFO message=Stream public-mytable is using incremental replication with replication key xmin cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
2023-08-25T17:06:34.041319Z [info ] time=2023-08-25 17:06:31 name=tap_postgres level=CRITICAL message='NoneType' object has no attribute 'get' cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
...
2023-08-25T17:06:34.043415Z [info ] replication_key_sql_datatype = md_map.get(('properties', replication_key)).get('sql-datatype') cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
2023-08-25T17:06:34.043529Z [info ] AttributeError: 'NoneType' object has no attribute 'get' cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the catalog populates metadata in md_map and how replication_key_sql_datatype looks up xmin, using the reported NoneType error as the entry point. Compare the current behavior with the linked Airbyte implementation, then verify that xmin can be selected as a replication column without the catalog lookup failing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100