MeltanoLabs / MeltanoLabs/tap-postgres
feature: Initialize new LOG_BASED streams by forcing an initial FULL_TABLE sync
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
When a stream with no prior bookmark is included in a LOG_BASED sync, the tap currently starts reading the replication slot from LSN 0 (the oldest retained WAL position). For an established slot this forces a full walk of all retained WAL for every registered table - even ones that already have bookmarks - before reaching the current tip. The user has no way to avoid this without manually pre-seeding the new stream with a FULL_TABLE sync.
Desired behavior
When the tap detects that one or more LOG_BASED streams have no prior LSN bookmark (i.e. they are new to log-based replication), it should automatically:
- Snapshot those streams via
FULL_TABLEreplication to capture their current state. - Record the WAL tip LSN at (or just before) the snapshot as each new stream's starting bookmark.
- Proceed with normal
LOG_BASEDstreaming for all streams, now all starting from a valid LSN.
This matches the pattern used by other CDC tools: Debezium incremental snapshots and Airbyte new-schema handling both require an initial snapshot at a known LSN before streaming begins.
Current workaround
Sync the new stream once with FULL_TABLE or INCREMENTAL replication, then switch it to LOG_BASED. See the README.
Implementation notes
- The LSN recorded as the new stream's bookmark should be captured before (or at the start of) the full-table read, not after, to avoid missing changes that arrive while the snapshot is in progress.
- Detecting "new to log-based" is straightforward:
get_starting_replication_key_value()returnsNonefor a stream with no prior bookmark. - The automatic full-table path should be opt-out-able (e.g. a config flag or catalog metadata key) for users who prefer the current behavior or manage the bootstrap themselves.
- The
SingleConnectionWALReaderintap_postgres/wal_reader.pycomputesglobal_start_lsn = min(start_lsn for all streams), so a single unbookmarked stream currently pulls that value to 0. Resolving the bookmark before the WAL reader constructed fixes this at the root.
Related
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with get_starting_replication_key_value() and the WAL-reader construction in tap_postgres/wal_reader.py, especially where global_start_lsn is calculated. Trace how FULL_TABLE replication and bookmarks are handled, then verify that new LOG_BASED streams are initialized before streaming and that the automatic path can be opted out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100