electric-sql / electric-sql/electric-circuits
Persistent shape catalog + lazy restart recovery
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Foundation for restart-safety (closes the root cause behind #3 and #4) and prerequisite for HTTP caching and retention work.
**Scope (aligned in discussion):**
- **Catalog:** persist `{shapeId, table, canonical predicate, columns, kind, backfill-complete, tailer offset, last_read}`. Proposed store: a `__meta/shapes` durable stream — keeps the engine node-stateless (all durable state = PG slot + DS log); boot folds the catalog stream only. `last_read` flushed periodically/batched, never per-read.
- **Lazy recovery — boot must be O(catalog), not O(shapes × stream folds).** With thousands of shapes, per-shape DS reads at boot are too slow. Boot does: load catalog index → register predicates into the shared routers (cheap, in-memory) → resume **per-table** tailers from the min persisted offset per table (one shared replay per table catches every shape up — no per-shape work). Expensive per-shape state is rebuilt **on first touch**: `/v1/shape` key sets by folding the shape stream on first unknown-handle request (no more restart 409 stampede), gates only if a backfill was incomplete.
- **Incomplete backfills are discarded** at boot (catalog row + stream removed), same policy as upstream (`shape_status.ex:54-81`).
- **Collision-free shape ids** across restarts (persist the counter or random ids) — fixes #3.
- **Orphan GC:** background sweep of DS `shape/*` streams not present in the catalog.
**Interactions:** table-stream trimming (future) must never trim below the min persisted tailer offset; dormant-shape reactivation (retention issue, TBD) reuses the same lazy catch-up path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start by tracing the /v1/shape entry point, the __meta/shapes durable stream, and the per-table tailer boot path; compare incomplete-backfill handling with shape_status.ex:54-81. Map how catalog loading, lazy first-touch recovery, collision-free IDs, and orphan GC fit together. Done means restart recovery is lazy and shared per table, incomplete backfills are discarded, IDs remain unique, and unreferenced shape streams are swept.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, rust
- Domain
- api, backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100