electric-sql / electric-sql/electric-circuits

Strategic: migrate ingest to streaming logical replication (pgoutput, proto v2+)

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30
Forks
6
PR merge metrics
No merged PRs in 30d

Description

The polling ingest (`pg_logical_slot_peek_changes` + `test_decoding`, 50ms interval) works but caps the architecture. Moving to the streaming replication protocol with `pgoutput` proto v2+ (PG 14+) subsumes several open problems at once:

- **Large transactions** (#6): proto v2 streams *in-progress* transactions — eliminates whole-txn buffering at the source (upstream Electric is still on proto v1 with a 250MiB hard limit and a spill TODO; this would leapfrog them).
- **DDL awareness** (#7): relation messages arrive in-stream — no polling reconciler needed for column changes.
- **Latency:** push instead of 50ms poll; keepalives + standby status updates give server-driven lag feedback and prevent `wal_sender_timeout` disconnects.
- **WAL retention hygiene** (#5): keepalive acks advance the slot during idle periods so it doesn't pin WAL on unrelated traffic.

Design notes: dedicated replication connection (session-stateful, like today's ingestor); ack (`flushed_wal` in standby updates) only after DS appends are durable — our current fsync-before-advance invariant maps directly onto the standby-update protocol; `test_decoding` parsing (`replication.rs:348-452`) is replaced by binary pgoutput decoding. Upstream references: `postgres/replication_client.ex` (+ `connection_setup.ex` state machine, `message_converter.ex`), keepalive interval `min(wal_sender_timeout/3, 15s)`.

Sequencing: after the P0 wave (#3–#8) — the guard+spill mitigations in #6 keep the polling path safe until this lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start by reading replication.rs:348-452 to understand the current test_decoding parser, then review the referenced upstream replication_client.ex, connection_setup.ex, and message_converter.ex. The work is complete when ingest uses streaming pgoutput proto v2+ on a dedicated replication connection, handles keepalives and relation messages, and advances flushed_wal only after durable appends.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.