erigontech / erigontech/erigon

execution/stagedsync (senders): ECDSA sender-recovery underutilises cores (~5/12) — serial feed/drain/flush bottleneck

Open
#21,637 1 comment 0 reactions 0 assignees View on GitHub
performance tech debt reduction
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Summary

The Senders stage (`execution/stagedsync/stage_senders.go`) spins up a full
`runtime.NumCPU()` pool of ECDSA recovery workers, but in practice only ~5 of 12
hardware threads are busy during a large sender-recovery run. The
embarrassingly-parallel part (pubkey recovery) is starved by the serial work
around it, so adding recovery cores doesn't help — it's a pipeline/Amdahl
imbalance, not a worker-count cap.

## Observed

During a mainnet backfill (~44k-block gap, `--prune.mode=minimal`) the senders
stage sat at ~477% CPU (≈5 cores) for an extended period on a 12-thread box,
repeatedly emitting `[3/6 Senders] Flushed buffer file erigon-sortable-buf-*`.

## Why it isn't a worker-count cap

`numOfGoroutines = secp256k1.NumOfContexts()` (stage_senders.go:63), and
`NumOfContexts()` returns `runtime.NumCPU()` (one secp256k1 context per CPU). So
the recovery pool is 12; each worker has its own crypto context
(stage_senders.go:108-115). The recovery itself can use all cores.

## Root cause — serial sections starve the parallel recovery

1. **Feed (main goroutine, serial):** reads each canonical block body (snapshot
mmap → page-fault IO) and RLP-decodes every tx before pushing recovery jobs.
Body read + decode is single-threaded and dominates the cheap per-tx ECDSA
recovery, so workers idle waiting to be fed.
2. **Drain (single goroutine + lock):** one goroutine drains `out`, reassembles
per-block sender arrays under `pendingMu`, and `collectorSenders.Collect()`s
(stage_senders.go:134-174). Serial chokepoint with lock contention.
3. **ETL spill flushes:** the collector uses `SmallSortableBuffers` with
`SortAndFlushInBackground(false)` (stage_senders.go:117-119); the periodic
spill flushes stall the pipeline (the `Flushed buffer file` lines).

## Suggested directions (not now)

- Parallelise / pipeline the body read + RLP-decode feed so the recovery pool
stays fed (e.g. a small reader/decoder pool, or prefetch+decode ahead).
- Reduce the single-drain bottleneck (shard the drain, or lock-free per-block
assembly).
- Larger ETL buffers and/or genuinely background flush so spill IO doesn't stall
the pipeline.

## Priority

Looks fixable but is not urgent — filing to track. Same shape as the broader
"parallel compute gated by serial IO/marshalling around it" pattern.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.