electric-sql / electric-sql/electric
durable-streams-rust: use the cold tier as the durable bulk store (make local per-stream files a non-durable read cache)
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Context
The write **cardinality cliff** is caused by the WAL checkpoint's `O(N_touched)` per-stream `fdatasync` storm (see investigation in `CARDINALITY_CLIFF_FIX.md`). The checkpoint exists only to make per-stream data durable so the WAL can be recycled.
## Proposal
The server already has a cold tier (S3/MinIO segments, `--tier-*`). Make the **local per-stream files a pure read cache, not a durability target**:
- Durability = WAL (recent) + cold-tier segments (bulk).
- The WAL recycles once its records are persisted to the tier — which is already **batched into bounded segments**, i.e. `O(segments)` uploads/fsyncs, not `O(streams)`.
- Recovery = load tier state + replay the WAL tail. The per-stream local files are rebuilt lazily as a cache and never need a per-file `fdatasync`.
This deletes the `O(N)` checkpoint by reusing infrastructure that already exists, and fits the tiered architecture better than one-file-per-stream local durability.
## Open questions
- Is the cold tier currently authoritative enough for recovery (upload durability, ordering vs WAL recycle floor)?
- Interaction with `--tier-segment-bytes` / `--tier-compact-bytes` cadence.
## Alternatives tracked separately
- Log-structured local store (#LOG_STRUCTURED_ISSUE).
- Incremental/staggered checkpoint + `--wal-checkpoint-syncfs` (being prototyped now).
Contributor guide
Research direction
Start with CARDINALITY_CLIFF_FIX.md and the WAL checkpoint and cold-tier entry points exposed by the --tier-* options. Trace how tier uploads, WAL recycling, and recovery ordering interact, including --tier-segment-bytes and --tier-compact-bytes. Done means the open recovery and durability questions are resolved and the proposed cache-only local files no longer require an O(N) durability checkpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100