electric-sql / electric-sql/electric

durable-streams-rust: log-structured per-stream data store to eliminate O(N) checkpoint fdatasync

Open
#4,695 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Context

Investigation of the write **cardinality cliff** (throughput collapsing from ~279k ops/s at 100 streams to ~11k at 100k streams) traced the root cause to the WAL **checkpoint**: it makes each touched stream's data file durable via **one `fdatasync` per stream** — `O(N_touched)` device barriers (measured ~29s at 200k streams). That storm saturates the disk and stalls the commit-path fsyncs, which is the cliff.

Multi-lane hardware does NOT fix it: on `c4d-standard-64-lssd` with one physical NVMe device mounted per shard, throughput at 100k streams was flat across 1/2/4/6 lanes (~10-11k), because all shards' checkpoints fire simultaneously (shared ticker) and saturate every device at once.

## Proposal

The `O(N)` cost exists because there is **one file per stream**. Move stream data into a **bounded set of shared append-only segments** (log-structured, like the WAL itself). Durability then costs `O(segments)` fsyncs, independent of stream count. Recovery replays/indexes the segments; a background compaction reclaims space via sequential rewrites + a handful of fsyncs.

- Pro: structurally eliminates the per-stream checkpoint storm; the cliff goes away.
- Con: per-stream reads become indexed segment lookups; requires a compaction path.

## Alternatives tracked separately
- Incremental/staggered checkpoint + `--wal-checkpoint-syncfs` (cheaper, amortize/coalesce the storm) — being prototyped now.
- Reuse the existing cold tier as the durable bulk store (#COLD_TIER_ISSUE).

Findings: `packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md`.

Contributor guide

Open the contributing guide

Research direction

Start with packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md and trace the WAL checkpoint path it describes. Define the segment layout, recovery/indexing behavior, and compaction path before implementation. Done means durability uses a bounded number of segment fsyncs rather than one fdatasync per touched stream, while reads and recovery remain functional.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.