electric-sql / electric-sql/electric-circuits

Unbounded memory on large transactions: peek cap escalates until the whole txn fits in RAM

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
30
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Found during the production-readiness investigation.

**Symptom / failure scenario:** `decode_batch` buffers a whole transaction in `tx_envs` to stamp commit lsn/xid/seq, and when a transaction doesn't fit in `PEEK_CAP` (5000 rows) the cap escalates ×4 repeatedly until it does (`apps/engine/src/replication.rs:76-166`). Under `REPLICA IDENTITY FULL` each change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (`engine.rs:1252-1276`) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.

**Fix direction (staged):**
1. **Guard now:** a max-txn-size limit that fails loudly (invalidate affected shapes → client refetch) instead of OOM. Upstream uses 250 MiB (`message_converter.ex:96-102`).
2. **Spill:** past a threshold, spill `tx_envs` to a temp file, stream through stamping on COMMIT.
3. **Real fix:** streaming logical replication with pgoutput proto v2+ (PG 14+), which streams in-progress large transactions — eliminates whole-txn buffering at the source (upstream hasn't done this either; their spill is still a TODO).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start in apps/engine/src/replication.rs:76-166 to trace decode_batch's PEAK_CAP growth and tx_envs buffering, then inspect the tailer's staging in engine.rs:1252-1276. Compare the proposed guard with upstream's message_converter.ex:96-102. Done should prevent large transactions from causing an OOM and define the affected-shape recovery behavior, or clearly scope the spill/streaming work separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.