commonwarexyz / commonwarexyz/constantinople

Indexer node OOMs during finalized-upload catch-up after restart under load

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
27
Forks
8
PR merge metrics
No merged PRs in 30d

Description

### Summary

The chain indexer secondary OOM-kills during catch-up whenever it restarts with a backlog of finalized blocks to upload. Memory climbs ~320 MB/s to a ~31 GB kill on a 32 GB node, then crash-loops, because the finalized-upload pipeline admits work bounded by item *count* while items are large. At ~100k txs/block each in-flight block holds ~300 MB across its concurrent representations (decoded queue entry, prepared QMDB/merkle ops, staged SQL rows, wire-encoded commit), so a full pipeline is a multi-GB commitment independent of store health.

### Environment

- `deploy.sh` config: c8a.4xlarge validators (16 vCPU / **32 GiB**), external chain store, `--spammer-accounts 49152`, `--max-propose-bytes 16777216`, ~280k TPS sustained, ~100k txs/block, ~163 blocks/min.
- Indexer is a secondary validator with the full upload pipeline (`upload_buffer`/count-bounded, default 64).

### Reproduction

1. Run a network at the above load with the indexer ingesting normally.
2. Restart the indexer (or let it OOM once) so a backlog accrues on the durable finalized-upload queue.
3. On restart it fills all ~64 count-bounded pipeline slots immediately; RSS climbs ~320 MB/s to ~31 GB and the kernel OOM-kills it. Because queue acks are gated on upload completion, each cycle makes little durable progress and it crash-loops.

### Investigation

- jemalloc heap profile at the moment of death showed ~30.7 GB **live** (in-use) heap: ~17 GB in the upload pipeline (entry decode, QMDB prepare, commit staging, protobuf encoding), ~5.4 GB marshal delivery, ~6 GB legitimate fixed caches. Genuine accumulation, not allocator churn.
- Confirmed count-bounded, not store-bound: the store's commit latency was flat ~23 ms throughout; the client-side pipeline was the constraint.

### Example solution shape

Byte-aware admission into the pipeline, prototyped on branch `michael/indexer-byte-admission` (commits `905c62d`, `301a46d`, `6775caf`):

- Charge a byte budget (default 3 GiB) on each entry's encoded size × amplification factor before decode; release only after the upload is acked. Backpressure parks the backlog on disk instead of in memory.
- Validated live: restart under full load caught up 2,037 blocks, zero kills, `reserved_bytes` pinned at the cap, peak RSS ~24 GB vs ~31 GB death.
- Note: the ~24 GB peak still includes a backlog-proportional marshal share; the pipeline budget bounds only the indexer-owned portion. The marshal's catch-up memory affects all validators and is a separate problem. Two review follow-ups remain: a benign spawn-order `WriterOutOfSync` retry, and documenting the amplification constant's provenance.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the prototype branch michael/indexer-byte-admission and commits 905c62d, 301a46d, and 6775caf, then reproduce the restart catch-up under load described above. Done means byte-aware admission prevents the indexer from OOM-crashing, reserves stay at the configured cap, and the remaining WriterOutOfSync retry and amplification-constant documentation follow-ups are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.