pathwaycom / pathwaycom/pathway

Support external replication slots in `pw.io.postgres.read`; add persistence support

Open
#222 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
62.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.

pw.io.postgres.read in "streaming" mode currently always creates a temporary replication slot internally. The slot is automatically dropped when the connection is closed. As stated in the documentation, this is an intentional guardrail against WAL accumulation. However, this design makes it impossible to use Pathway's persistence mechanism with the Postgres connector: since the slot is gone after restart, the connector cannot resume from where it left off and must perform a full re-snapshot on every startup.

Describe the solution you'd like

Add support for an externally managed (pre-created) replication slot in pw.io.postgres.read. When the user provides a slot name via a new parameter (e.g. replication_slot_name), the connector should:

  1. Use that slot as-is instead of creating a temporary one.
  2. Integrate with Pathway's persistence layer by storing the LSN of the last acknowledged WAL record as the connector's persistent offset. On restart, the connector should resume reading from that LSN rather than re-snapshotting.
  3. Emit a warning at startup that the lifecycle of the replication slot (creation, monitoring, and deletion) is the user's responsibility, and that an inactive or abandoned slot can cause unbounded WAL retention on the PostgreSQL server.

The existing behavior (temporary slot, no persistence) should remain the default when no slot name is provided.

Describe alternatives you've considered

  • Using pw.io.debezium.read with Kafka as a persistence-capable alternative — works, but introduces significant operational overhead (Kafka + Debezium) for use cases where a direct Postgres connection would suffice.
  • Absorbing the re-snapshot cost: acceptable for small tables but not for large ones where a full re-read on every restart is prohibitive.

Additional context

The documentation for pw.io.postgres.read currently contains the following note:

"There is no need to create a replication slot manually, and doing so is strongly discouraged."

This feature request proposes making the externally-managed slot an opt-in mode with explicit user acknowledgment (via the warning), rather than removing the guardrail from the default path. The LSN-based offset aligns naturally with how PostgreSQL logical replication tracks progress via confirmed_flush_lsn in pg_replication_slots.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the pw.io.postgres.read entry point and review its streaming-mode documentation, including the replication-slot warning. Trace how the connector currently handles its temporary slot and persistence state. Done means an opt-in external slot resumes from the persisted acknowledged LSN, emits the requested lifecycle warning, and preserves the current default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.