electric-sql / electric-sql/electric

durable-streams-rust: use the cold tier as the durable bulk store (make local per-stream files a non-durable read cache)

Open
#4,696 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Context

The write **cardinality cliff** is caused by the WAL checkpoint's `O(N_touched)` per-stream `fdatasync` storm (see investigation in `CARDINALITY_CLIFF_FIX.md`). The checkpoint exists only to make per-stream data durable so the WAL can be recycled.

## Proposal

The server already has a cold tier (S3/MinIO segments, `--tier-*`). Make the **local per-stream files a pure read cache, not a durability target**:

- Durability = WAL (recent) + cold-tier segments (bulk).
- The WAL recycles once its records are persisted to the tier — which is already **batched into bounded segments**, i.e. `O(segments)` uploads/fsyncs, not `O(streams)`.
- Recovery = load tier state + replay the WAL tail. The per-stream local files are rebuilt lazily as a cache and never need a per-file `fdatasync`.

This deletes the `O(N)` checkpoint by reusing infrastructure that already exists, and fits the tiered architecture better than one-file-per-stream local durability.

## Open questions
- Is the cold tier currently authoritative enough for recovery (upload durability, ordering vs WAL recycle floor)?
- Interaction with `--tier-segment-bytes` / `--tier-compact-bytes` cadence.

## Alternatives tracked separately
- Log-structured local store (#LOG_STRUCTURED_ISSUE).
- Incremental/staggered checkpoint + `--wal-checkpoint-syncfs` (being prototyped now).

Contributor guide

Open the contributing guide

Research direction

Start with CARDINALITY_CLIFF_FIX.md and the WAL checkpoint and cold-tier entry points exposed by the --tier-* options. Trace how tier uploads, WAL recycling, and recovery ordering interact, including --tier-segment-bytes and --tier-compact-bytes. Done means the open recovery and durability questions are resolved and the proposed cache-only local files no longer require an O(N) durability checkpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.