electric-sql / electric-sql/electric-circuits
DS disk growth: table-stream prefix trimming (log-native in ds-rust), size accounting; no compaction
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Today `table/` streams accumulate every change forever, the engine sets no TTLs, and orphaned `shape/*` streams from previous runs are never reclaimed. Unbounded disk on any long-running deployment.
**Decisions (aligned in discussion):** production DS target is **electric-ds-rust**; primitives are **log-native** there; **no compaction** — shape streams are never key-merged.
**Scope:**
1. **`table/*` prefix trimming (ds-rust primitive):** the engine publishes a per-table low watermark = `min(active tailer offsets, dormant shapes' resume offsets)`; the DS server reclaims bytes below it (segment drop). Reads below the watermark answer with a distinguishable error so a resuming consumer knows to re-backfill. Interaction with #9: trimming past a dormant shape's offset either forces its eviction first or degrades its reactivation to a fresh PG backfill (acceptable path — pick one and document).
2. **Per-stream size accounting (ds-rust):** expose stream byte sizes (list/HEAD) — needed by #9's disk budget and for the trim watermark's effectiveness metrics.
3. **Orphan GC:** covered in #8 (catalog sweep) — listed here for completeness.
4. **Shape-stream growth without compaction:** bounded by #9 eviction. For hot, long-lived shapes whose log grows far beyond their live set, add **rotation**: when `stream bytes / live-set bytes` exceeds a threshold, drop and recreate the shape stream (fresh backfill fold), clients get `must-refetch`. Reuses existing invalidation paths; no merge machinery, no compaction-boundary offset semantics.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with the production electric-ds-rust target and its log-native primitives, then read the related decisions in #9 and #8. Map the table-stream watermark, per-stream size reporting, and shape-stream rotation entry points before choosing the dormant-shape behavior. Done means trimming, size accounting, and rotation are documented and validated without adding compaction or merge semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100