MeltanoLabs / MeltanoLabs/tap-postgres

New tables without replication_key column breaks replication

Open
#810 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
26
Forks
42
PR merge metrics
No merged PRs in 30d

Description

We apply the following config so that we automatically replicate new tables:

      select:
        - '*.*'
      metadata:
        '*':
          replication-key: updated_at
          updated_at:
            is-replication-key: true

If a new table does not have updated_at column, the whole tap errors out and our pipeline stays broken until we add a manual override:

        public-table:
          replication-key: created_at
          created_at:
            is-replication-key: true

        public-other_table:
          replication-method: FULL_TABLE
          replication-key: ''

We should be able to configure a fallback behavior in these cases, such as a warning

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by tracing how select and metadata replication-key settings are applied when a newly discovered table lacks updated_at. Define and test a fallback that warns or selects an alternate replication behavior without stopping the tap, including the existing manual override cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
data, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.