electric-sql / electric-sql/electric-circuits
Failover/slot loss unhandled: reconnect loop never re-checks the slot, no timeline detection
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Found during the production-readiness investigation.
**Symptom / failure scenario:** the ingestor's reconnect loop (`apps/engine/src/replication.rs:42-56`) retries `pg::connect` with a flat 500ms sleep and resumes peeking — `ensure_slot` runs only in `setup_postgres` at boot. After a failover to a promoted replica (or a dropped/invalidated slot), the slot doesn't exist: the loop errors forever. If the process is instead rebooted, `ensure_slot` silently creates a **fresh slot at the new head LSN**, losing every change in the gap — shapes are silently missing data with no signal.
There is also no timeline/system-id detection (upstream persists `{pg_system_identifier, timeline_id}` and purges all shapes on mismatch), and no exponential backoff.
**Fix direction:** persist system-id/timeline and compare at (re)connect → on mismatch or slot-gone, purge all shapes and force client refetch rather than continuing; exponential backoff with jitter; export a retained-WAL gauge. Reference: sync-service `timeline.ex`, `connection/manager.ex:452-500`, `db_connection_error.ex`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with apps/engine/src/replication.rs:42-56 and trace setup_postgres and the reconnect loop. Compare the proposed behavior with timeline.ex, connection/manager.ex:452-500, and db_connection_error.ex. Done means reconnects detect system-id, timeline, or slot loss, purge shapes and force refetches, use backoff with jitter, and export a retained-WAL gauge.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- databases, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100