electric-sql / electric-sql/electric-circuits
Unbounded memory on large transactions: peek cap escalates until the whole txn fits in RAM
- Dominant language
- Rust
- Stars
- 30
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Found during the production-readiness investigation.
**Symptom / failure scenario:** `decode_batch` buffers a whole transaction in `tx_envs` to stamp commit lsn/xid/seq, and when a transaction doesn't fit in `PEEK_CAP` (5000 rows) the cap escalates ×4 repeatedly until it does (`apps/engine/src/replication.rs:76-166`). Under `REPLICA IDENTITY FULL` each change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (`engine.rs:1252-1276`) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.
**Fix direction (staged):**
1. **Guard now:** a max-txn-size limit that fails loudly (invalidate affected shapes → client refetch) instead of OOM. Upstream uses 250 MiB (`message_converter.ex:96-102`).
2. **Spill:** past a threshold, spill `tx_envs` to a temp file, stream through stamping on COMMIT.
3. **Real fix:** streaming logical replication with pgoutput proto v2+ (PG 14+), which streams in-progress large transactions — eliminates whole-txn buffering at the source (upstream hasn't done this either; their spill is still a TODO).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start in apps/engine/src/replication.rs:76-166 to trace decode_batch's PEAK_CAP growth and tx_envs buffering, then inspect the tailer's staging in engine.rs:1252-1276. Compare the proposed guard with upstream's message_converter.ex:96-102. Done should prevent large transactions from causing an OOM and define the affected-shape recovery behavior, or clearly scope the spill/streaming work separately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100