electric-sql / electric-sql/electric
durable-streams-rust: log-structured per-stream data store to eliminate O(N) checkpoint fdatasync
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Context
Investigation of the write **cardinality cliff** (throughput collapsing from ~279k ops/s at 100 streams to ~11k at 100k streams) traced the root cause to the WAL **checkpoint**: it makes each touched stream's data file durable via **one `fdatasync` per stream** — `O(N_touched)` device barriers (measured ~29s at 200k streams). That storm saturates the disk and stalls the commit-path fsyncs, which is the cliff.
Multi-lane hardware does NOT fix it: on `c4d-standard-64-lssd` with one physical NVMe device mounted per shard, throughput at 100k streams was flat across 1/2/4/6 lanes (~10-11k), because all shards' checkpoints fire simultaneously (shared ticker) and saturate every device at once.
## Proposal
The `O(N)` cost exists because there is **one file per stream**. Move stream data into a **bounded set of shared append-only segments** (log-structured, like the WAL itself). Durability then costs `O(segments)` fsyncs, independent of stream count. Recovery replays/indexes the segments; a background compaction reclaims space via sequential rewrites + a handful of fsyncs.
- Pro: structurally eliminates the per-stream checkpoint storm; the cliff goes away.
- Con: per-stream reads become indexed segment lookups; requires a compaction path.
## Alternatives tracked separately
- Incremental/staggered checkpoint + `--wal-checkpoint-syncfs` (cheaper, amortize/coalesce the storm) — being prototyped now.
- Reuse the existing cold tier as the durable bulk store (#COLD_TIER_ISSUE).
Findings: `packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md`.
Contributor guide
Research direction
Start with packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md and trace the WAL checkpoint path it describes. Define the segment layout, recovery/indexing behavior, and compaction path before implementation. Done means durability uses a bounded number of segment fsyncs rather than one fdatasync per touched stream, while reads and recovery remain functional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100