electric-sql / electric-sql/electric-circuits

Failover/slot loss unhandled: reconnect loop never re-checks the slot, no timeline detection

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
30
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Found during the production-readiness investigation.

**Symptom / failure scenario:** the ingestor's reconnect loop (`apps/engine/src/replication.rs:42-56`) retries `pg::connect` with a flat 500ms sleep and resumes peeking — `ensure_slot` runs only in `setup_postgres` at boot. After a failover to a promoted replica (or a dropped/invalidated slot), the slot doesn't exist: the loop errors forever. If the process is instead rebooted, `ensure_slot` silently creates a **fresh slot at the new head LSN**, losing every change in the gap — shapes are silently missing data with no signal.

There is also no timeline/system-id detection (upstream persists `{pg_system_identifier, timeline_id}` and purges all shapes on mismatch), and no exponential backoff.

**Fix direction:** persist system-id/timeline and compare at (re)connect → on mismatch or slot-gone, purge all shapes and force client refetch rather than continuing; exponential backoff with jitter; export a retained-WAL gauge. Reference: sync-service `timeline.ex`, `connection/manager.ex:452-500`, `db_connection_error.ex`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with apps/engine/src/replication.rs:42-56 and trace setup_postgres and the reconnect loop. Compare the proposed behavior with timeline.ex, connection/manager.ex:452-500, and db_connection_error.ex. Done means reconnects detect system-id, timeline, or slot loss, purge shapes and force refetches, use backoff with jitter, and export a retained-WAL gauge.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
databases, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.