electric-sql / electric-sql/electric-circuits

Ops hardening: graceful shutdown, readiness probe, production metrics

Open
#16 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30
Forks
6
PR merge metrics
No merged PRs in 30d

Description

**Graceful shutdown (engine):** `axum::serve` has no `with_graceful_shutdown` and no SIGTERM handler (`apps/engine/src/main.rs:67`) — the process is hard-killed, dropping in-flight appends and long-polls (recovered only via at-least-once re-peek). Add SIGTERM handling: stop accepting, complete in-flight appends, close long-polls cleanly, stop the ingestor after its current batch (post-advance). The TS ds-server/api-server already handle SIGTERM.

**Readiness vs liveness:** `/health` returns a static "ok" before Postgres or DS are reachable (`http.rs:16`). Add `/ready` gated on {PG connected, slot exists and streaming, DS reachable}; keep `/health` as pure liveness. Optionally upstream's hold-briefly-then-503 pattern for requests arriving during startup (5s, then `retry-after`).

**Metrics** (gaps vs the set upstream exports; ours cover engine internals well but not ops):
- `slot_retained_wal_size` + `confirmed_flush_lsn_lag` (bytes) — the two that page you before the primary's disk fills
- replication receive-lag distribution (commit-to-ingest ms)
- request counts by status (`/v1/shape` + control plane), tagged known-error vs unexpected
- backfill duration/bytes/rows histograms; per-shape append-failure counter
- active vs total shapes (with the #9 tiers: active/dormant), DS disk usage per class (`table/*` vs `shape/*`)
- error-rate counters on ingest decode/append paths

These land on the utility port (see auth issue). Wire the existing `/metrics/prometheus` exporter; add OTel trace-sampling knobs before enabling tracing at production volume (upstream defaults to 1% span sampling).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with apps/engine/src/main.rs:67 and http.rs:16, then compare the existing TS ds-server/api-server SIGTERM handling and the /metrics/prometheus exporter. Break the work into shutdown, readiness, and production metrics, confirming the required PostgreSQL, replication, DS, request, backfill, shape, and ingest signals. Done means graceful SIGTERM behavior, distinct /health and gated /ready endpoints, and the listed metrics and OTel sampling controls on the utility port.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust, typescript
Domain
backend, databases, infrastructure, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.