electric-sql / electric-sql/electric-circuits
Ops hardening: graceful shutdown, readiness probe, production metrics
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
**Graceful shutdown (engine):** `axum::serve` has no `with_graceful_shutdown` and no SIGTERM handler (`apps/engine/src/main.rs:67`) — the process is hard-killed, dropping in-flight appends and long-polls (recovered only via at-least-once re-peek). Add SIGTERM handling: stop accepting, complete in-flight appends, close long-polls cleanly, stop the ingestor after its current batch (post-advance). The TS ds-server/api-server already handle SIGTERM.
**Readiness vs liveness:** `/health` returns a static "ok" before Postgres or DS are reachable (`http.rs:16`). Add `/ready` gated on {PG connected, slot exists and streaming, DS reachable}; keep `/health` as pure liveness. Optionally upstream's hold-briefly-then-503 pattern for requests arriving during startup (5s, then `retry-after`).
**Metrics** (gaps vs the set upstream exports; ours cover engine internals well but not ops):
- `slot_retained_wal_size` + `confirmed_flush_lsn_lag` (bytes) — the two that page you before the primary's disk fills
- replication receive-lag distribution (commit-to-ingest ms)
- request counts by status (`/v1/shape` + control plane), tagged known-error vs unexpected
- backfill duration/bytes/rows histograms; per-shape append-failure counter
- active vs total shapes (with the #9 tiers: active/dormant), DS disk usage per class (`table/*` vs `shape/*`)
- error-rate counters on ingest decode/append paths
These land on the utility port (see auth issue). Wire the existing `/metrics/prometheus` exporter; add OTel trace-sampling knobs before enabling tracing at production volume (upstream defaults to 1% span sampling).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with apps/engine/src/main.rs:67 and http.rs:16, then compare the existing TS ds-server/api-server SIGTERM handling and the /metrics/prometheus exporter. Break the work into shutdown, readiness, and production metrics, confirming the required PostgreSQL, replication, DS, request, backfill, shape, and ingest signals. Done means graceful SIGTERM behavior, distinct /health and gated /ready endpoints, and the listed metrics and OTel sampling controls on the utility port.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust, typescript
- Domain
- backend, databases, infrastructure, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100