erigontech / erigontech/erigon
Very slow BlobHistoryDownloader (~0.04 slots/s) and missing historical Fulu blobs
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 465
Description
### System information
Erigon version: `erigon version 3.6.1-0c4d9c91`
OS & Version: `Linux`
Erigon Command (with flags/config): /opt/ethmain/core/erigon --chain=mainnet --networkid=1 --datadir=/mnt/ethmain/node --prune.mode=archive --torrent.download.rate=64mb --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.api=admin,eth,net,web3,debug,trace,txpool,erigon --trace.maxtraces=2000 --ws --ws.port=8546 --rpc.batch.limit=10000 --rpc.returndata.limit=1000000 --rpc.gascap=0 --rpc.txfeecap=0 --beacon.api.addr=0.0.0.0 --be acon.api.port=3500 --beacon.api.cors.allow-origins=* --beacon.api=beacon,builder,config,debug,events,node,validator,lighthouse --port=30303 --nodekey=/mnt/ethmain/node/nodekey --caplin.blobs-immediate-backfill --caplin.states-archive --ca plin.blocks-archive --caplin.blobs-archive --caplin.blobs-no-pruning
Consensus Layer: `Caplin`
Consensus Layer Command (with flags/config): No
Chain/Network: `1`
### Summary
My node follows the chain head normally, but historical Blob Sidecars are missing. After upgrading to v3.6.1, BlobHistoryDownloader is running, but its backward scan advances at only approximately 0.03–0.04 slots/s.
Could you help confirm whether this behavior is expected and how to accelerate historical P2P blob backfilling?
### Backfill logs
```bash
[INFO] [09-10|10:43:34.236] [BlobHistoryDownloader] Downloading blobs backwards slot=15181681 to=8626176 slots/sec=0.03 progress=0.0% eta=54631h45m8s
[INFO] [09-10|10:47:04.246] [BlobHistoryDownloader] Downloading blobs backwards slot=15181673 to=8626176 slots/sec=0.04 progress=0.0% eta=47802h44m46s
[INFO] [09-10|10:50:04.256] [BlobHistoryDownloader] Downloading blobs backwards slot=15181665 to=8626176 slots/sec=0.04 progress=0.0% eta=40974h2m55s
```
### Snapshot and hot-storage coverage
The snapshot directory contains 605 blobsidecars.seg files. Their filename ranges are contiguous:
[8,620,000, 14,670,000)
This is a filename-range check, not a verification of snapshot contents or loaded indexes.
The hot-storage directory is:
```bash
/mnt/ethmain/node/caplin/blobs/
```
It contains numeric buckets 1474 through 1518. Buckets 1467 through 1473 are absent, suggesting a potential gap after the snapshot boundary.
However, existing buckets also appear to have missing data; directory presence does not imply complete coverage.
### Confirmed API discrepancy
On this same node, slot 14740001 has 9 KZG commitments:
```bash
curl -s http://127.0.0.1:3500/eth/v2/beacon/blocks/14740001 |
jq '{
version,
slot: .data.message.slot,
commitments: (.data.message.body.blob_kzg_commitments | length),
finalized,
execution_optimistic
}'
```
Response:
```bash
{
"version": "fulu",
"slot": "14740001",
"commitments": 9,
"finalized": true,
"execution_optimistic": false
}
```
But the Blob Sidecar endpoint returns an empty array:
```bash
curl -s http://127.0.0.1:3500/eth/v1/beacon/blob_sidecars/14740001
{
"data": [],
"execution_optimistic": false,
"finalized": true,
"version": "fulu"
}
```
### Questions
Is this backfill speed expected for Fulu blocks? Could repeated PeerDAS column-recovery timeouts explain it, and which logs or metrics would confirm that?
Is there a supported way to accelerate backfilling—for example, recovery concurrency, peer selection, traffic limits, or connections to peers retaining historical columns?
Would enabling --caplin.snapgen help in this situation, or would snapshot generation remain blocked until missing sidecars are recovered? Should I avoid restarting while the current backward scan is in progress?
I would appreciate guidance on whether the node is behaving as expected and the recommended next steps to restore complete historical Blob availability.
Contributor guide
Assessment
This issue has not been assessed yet.