electric-sql / electric-sql/electric-circuits
Strategic: migrate ingest to streaming logical replication (pgoutput, proto v2+)
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
The polling ingest (`pg_logical_slot_peek_changes` + `test_decoding`, 50ms interval) works but caps the architecture. Moving to the streaming replication protocol with `pgoutput` proto v2+ (PG 14+) subsumes several open problems at once:
- **Large transactions** (#6): proto v2 streams *in-progress* transactions — eliminates whole-txn buffering at the source (upstream Electric is still on proto v1 with a 250MiB hard limit and a spill TODO; this would leapfrog them).
- **DDL awareness** (#7): relation messages arrive in-stream — no polling reconciler needed for column changes.
- **Latency:** push instead of 50ms poll; keepalives + standby status updates give server-driven lag feedback and prevent `wal_sender_timeout` disconnects.
- **WAL retention hygiene** (#5): keepalive acks advance the slot during idle periods so it doesn't pin WAL on unrelated traffic.
Design notes: dedicated replication connection (session-stateful, like today's ingestor); ack (`flushed_wal` in standby updates) only after DS appends are durable — our current fsync-before-advance invariant maps directly onto the standby-update protocol; `test_decoding` parsing (`replication.rs:348-452`) is replaced by binary pgoutput decoding. Upstream references: `postgres/replication_client.ex` (+ `connection_setup.ex` state machine, `message_converter.ex`), keepalive interval `min(wal_sender_timeout/3, 15s)`.
Sequencing: after the P0 wave (#3–#8) — the guard+spill mitigations in #6 keep the polling path safe until this lands.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start by reading replication.rs:348-452 to understand the current test_decoding parser, then review the referenced upstream replication_client.ex, connection_setup.ex, and message_converter.ex. The work is complete when ingest uses streaming pgoutput proto v2+ on a dedicated replication connection, handles keepalives and relation messages, and advances flushed_wal only after durable appends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100