commonwarexyz / commonwarexyz/constantinople
Indexer node OOMs during finalized-upload catch-up after restart under load
- Dominant language
- Rust
- Stars
- 27
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
The chain indexer secondary OOM-kills during catch-up whenever it restarts with a backlog of finalized blocks to upload. Memory climbs ~320 MB/s to a ~31 GB kill on a 32 GB node, then crash-loops, because the finalized-upload pipeline admits work bounded by item *count* while items are large. At ~100k txs/block each in-flight block holds ~300 MB across its concurrent representations (decoded queue entry, prepared QMDB/merkle ops, staged SQL rows, wire-encoded commit), so a full pipeline is a multi-GB commitment independent of store health.
### Environment
- `deploy.sh` config: c8a.4xlarge validators (16 vCPU / **32 GiB**), external chain store, `--spammer-accounts 49152`, `--max-propose-bytes 16777216`, ~280k TPS sustained, ~100k txs/block, ~163 blocks/min.
- Indexer is a secondary validator with the full upload pipeline (`upload_buffer`/count-bounded, default 64).
### Reproduction
1. Run a network at the above load with the indexer ingesting normally.
2. Restart the indexer (or let it OOM once) so a backlog accrues on the durable finalized-upload queue.
3. On restart it fills all ~64 count-bounded pipeline slots immediately; RSS climbs ~320 MB/s to ~31 GB and the kernel OOM-kills it. Because queue acks are gated on upload completion, each cycle makes little durable progress and it crash-loops.
### Investigation
- jemalloc heap profile at the moment of death showed ~30.7 GB **live** (in-use) heap: ~17 GB in the upload pipeline (entry decode, QMDB prepare, commit staging, protobuf encoding), ~5.4 GB marshal delivery, ~6 GB legitimate fixed caches. Genuine accumulation, not allocator churn.
- Confirmed count-bounded, not store-bound: the store's commit latency was flat ~23 ms throughout; the client-side pipeline was the constraint.
### Example solution shape
Byte-aware admission into the pipeline, prototyped on branch `michael/indexer-byte-admission` (commits `905c62d`, `301a46d`, `6775caf`):
- Charge a byte budget (default 3 GiB) on each entry's encoded size × amplification factor before decode; release only after the upload is acked. Backpressure parks the backlog on disk instead of in memory.
- Validated live: restart under full load caught up 2,037 blocks, zero kills, `reserved_bytes` pinned at the cap, peak RSS ~24 GB vs ~31 GB death.
- Note: the ~24 GB peak still includes a backlog-proportional marshal share; the pipeline budget bounds only the indexer-owned portion. The marshal's catch-up memory affects all validators and is a separate problem. Two review follow-ups remain: a benign spawn-order `WriterOutOfSync` retry, and documenting the amplification constant's provenance.
Contributor guide
Research direction
Start by reviewing the prototype branch michael/indexer-byte-admission and commits 905c62d, 301a46d, and 6775caf, then reproduce the restart catch-up under load described above. Done means byte-aware admission prevents the indexer from OOM-crashing, reserves stay at the configured cap, and the remaining WriterOutOfSync retry and amplification-constant documentation follow-ups are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100