MeltanoLabs / MeltanoLabs/tap-postgres

feature: Initialize new LOG_BASED streams by forcing an initial FULL_TABLE sync

Open
#799 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
Python
Stars
26
Forks
42
PR merge metrics
No merged PRs in 30d

Description

When a stream with no prior bookmark is included in a LOG_BASED sync, the tap currently starts reading the replication slot from LSN 0 (the oldest retained WAL position). For an established slot this forces a full walk of all retained WAL for every registered table - even ones that already have bookmarks - before reaching the current tip. The user has no way to avoid this without manually pre-seeding the new stream with a FULL_TABLE sync.

Desired behavior

When the tap detects that one or more LOG_BASED streams have no prior LSN bookmark (i.e. they are new to log-based replication), it should automatically:

  1. Snapshot those streams via FULL_TABLE replication to capture their current state.
  2. Record the WAL tip LSN at (or just before) the snapshot as each new stream's starting bookmark.
  3. Proceed with normal LOG_BASED streaming for all streams, now all starting from a valid LSN.

This matches the pattern used by other CDC tools: Debezium incremental snapshots and Airbyte new-schema handling both require an initial snapshot at a known LSN before streaming begins.

Current workaround

Sync the new stream once with FULL_TABLE or INCREMENTAL replication, then switch it to LOG_BASED. See the README.

Implementation notes

  • The LSN recorded as the new stream's bookmark should be captured before (or at the start of) the full-table read, not after, to avoid missing changes that arrive while the snapshot is in progress.
  • Detecting "new to log-based" is straightforward: get_starting_replication_key_value() returns None for a stream with no prior bookmark.
  • The automatic full-table path should be opt-out-able (e.g. a config flag or catalog metadata key) for users who prefer the current behavior or manage the bootstrap themselves.
  • The SingleConnectionWALReader in tap_postgres/wal_reader.py computes global_start_lsn = min(start_lsn for all streams), so a single unbookmarked stream currently pulls that value to 0. Resolving the bookmark before the WAL reader constructed fixes this at the root.

Related

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with get_starting_replication_key_value() and the WAL-reader construction in tap_postgres/wal_reader.py, especially where global_start_lsn is calculated. Trace how FULL_TABLE replication and bookmarks are handled, then verify that new LOG_BASED streams are initialized before streaming and that the automatic path can be opted out.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.