MeltanoLabs / MeltanoLabs/tap-postgres

feat: potentially add xmin replication?

Open
#219 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26
Forks
42
PR merge metrics
No merged PRs in 30d

Description

Airbyte has this as of July 10, see here. Haven't spent time scoping this out but doesnt work on current version of this plugin. I suspect this'd be easy to add, I assume the catalog is excluding hidden/system columns and it could just include them(?)

Specifically, I tried to just set xmin as the replication column, and it looks like it errors saying hey that column isnt in the catalog.

2023-08-25T17:06:34.041196Z [info     ] time=2023-08-25 17:06:31 name=tap_postgres level=INFO message=Stream public-mytable is using incremental replication with replication key xmin cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
2023-08-25T17:06:34.041319Z [info     ] time=2023-08-25 17:06:31 name=tap_postgres level=CRITICAL message='NoneType' object has no attribute 'get' cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
...
2023-08-25T17:06:34.043415Z [info     ]     replication_key_sql_datatype = md_map.get(('properties', replication_key)).get('sql-datatype') cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0
2023-08-25T17:06:34.043529Z [info     ] AttributeError: 'NoneType' object has no attribute 'get' cmd_type=elb consumer=False name=pipeline-0 producer=True stdio=stderr string_id=pipeline-0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the catalog populates metadata in md_map and how replication_key_sql_datatype looks up xmin, using the reported NoneType error as the entry point. Compare the current behavior with the linked Airbyte implementation, then verify that xmin can be selected as a replication column without the catalog lookup failing.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.