electric-sql / electric-sql/electric

durable-streams-rust: lazy stream-file fd management (1 fd per live stream ≈ nofile ceiling at 1M streams)

Open
#4,706 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

The WAL/data design holds one open fd per live stream for its lifetime (`Shared.file`, opened at create/recover). Measured on the wal-1m-diag run (2026-07-13): 1,005,724 fds at 1M streams — 96% of the default container hard limit (1,048,576). It was NOT the cause of the 1M throughput wall (that was checkpoint writeback on one device — fixed by `--stream-lanes`, #4705), but it is a hard scale ceiling just above 1M streams: the next stream open or socket accept hits EMFILE.

Options:
- Idle-close sweeper + on-demand reopen (`Shared.file: Option>`), bounded by an LRU-ish cap: hot streams keep fds, idle ones close. Touches the read path (`resolve_range` clones the Arc), append path (`ap.file`), and compaction swap — the Arc refcount model makes eviction safe (in-flight sendfile keeps the fd alive).
- Deployment mitigation meanwhile: raise LimitNOFILE on the node/containerd.

Related: #4695 (log-structured store) would remove per-stream files entirely; #4705 measured the residual per-file writeback slope (374k @100k → 212k @1M on 3 lanes).

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Shared.file is opened at create/recover, then read the paths named in the issue: resolve_range, ap.file, and compaction swap. Compare the idle-close/reopen option with the Arc refcount behavior and the related issues before choosing an approach. Done means live streams no longer require one open fd each while in-flight reads remain safe.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.