MeltanoLabs / MeltanoLabs/tap-postgres
New tables without replication_key column breaks replication
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
We apply the following config so that we automatically replicate new tables:
select:
- '*.*'
metadata:
'*':
replication-key: updated_at
updated_at:
is-replication-key: true
If a new table does not have updated_at column, the whole tap errors out and our pipeline stays broken until we add a manual override:
public-table:
replication-key: created_at
created_at:
is-replication-key: true
public-other_table:
replication-method: FULL_TABLE
replication-key: ''
We should be able to configure a fallback behavior in these cases, such as a warning
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing how select and metadata replication-key settings are applied when a newly discovered table lacks updated_at. Define and test a fallback that warns or selects an alternate replication behavior without stopping the tap, including the existing manual override cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100