electric-sql / electric-sql/electric
durable-streams-rust: lazy stream-file fd management (1 fd per live stream ≈ nofile ceiling at 1M streams)
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
The WAL/data design holds one open fd per live stream for its lifetime (`Shared.file`, opened at create/recover). Measured on the wal-1m-diag run (2026-07-13): 1,005,724 fds at 1M streams — 96% of the default container hard limit (1,048,576). It was NOT the cause of the 1M throughput wall (that was checkpoint writeback on one device — fixed by `--stream-lanes`, #4705), but it is a hard scale ceiling just above 1M streams: the next stream open or socket accept hits EMFILE.
Options:
- Idle-close sweeper + on-demand reopen (`Shared.file: Option>`), bounded by an LRU-ish cap: hot streams keep fds, idle ones close. Touches the read path (`resolve_range` clones the Arc), append path (`ap.file`), and compaction swap — the Arc refcount model makes eviction safe (in-flight sendfile keeps the fd alive).
- Deployment mitigation meanwhile: raise LimitNOFILE on the node/containerd.
Related: #4695 (log-structured store) would remove per-stream files entirely; #4705 measured the residual per-file writeback slope (374k @100k → 212k @1M on 3 lanes).
Contributor guide
Research direction
Start by tracing how Shared.file is opened at create/recover, then read the paths named in the issue: resolve_range, ap.file, and compaction swap. Compare the idle-close/reopen option with the Arc refcount behavior and the related issues before choosing an approach. Done means live streams no longer require one open fd each while in-flight reads remain safe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100