electric-sql / electric-sql/electric-circuits

DS disk growth: table-stream prefix trimming (log-native in ds-rust), size accounting; no compaction

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Today `table/` streams accumulate every change forever, the engine sets no TTLs, and orphaned `shape/*` streams from previous runs are never reclaimed. Unbounded disk on any long-running deployment.

**Decisions (aligned in discussion):** production DS target is **electric-ds-rust**; primitives are **log-native** there; **no compaction** — shape streams are never key-merged.

**Scope:**
1. **`table/*` prefix trimming (ds-rust primitive):** the engine publishes a per-table low watermark = `min(active tailer offsets, dormant shapes' resume offsets)`; the DS server reclaims bytes below it (segment drop). Reads below the watermark answer with a distinguishable error so a resuming consumer knows to re-backfill. Interaction with #9: trimming past a dormant shape's offset either forces its eviction first or degrades its reactivation to a fresh PG backfill (acceptable path — pick one and document).
2. **Per-stream size accounting (ds-rust):** expose stream byte sizes (list/HEAD) — needed by #9's disk budget and for the trim watermark's effectiveness metrics.
3. **Orphan GC:** covered in #8 (catalog sweep) — listed here for completeness.
4. **Shape-stream growth without compaction:** bounded by #9 eviction. For hot, long-lived shapes whose log grows far beyond their live set, add **rotation**: when `stream bytes / live-set bytes` exceeds a threshold, drop and recreate the shape stream (fresh backfill fold), clients get `must-refetch`. Reuses existing invalidation paths; no merge machinery, no compaction-boundary offset semantics.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with the production electric-ds-rust target and its log-native primitives, then read the related decisions in #9 and #8. Map the table-stream watermark, per-stream size reporting, and shape-stream rotation entry points before choosing the dormant-shape behavior. Done means trimming, size accounting, and rotation are documented and validated without adding compaction or merge semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.