erigontech / erigontech/erigon

Very slow BlobHistoryDownloader (~0.04 slots/s) and missing historical Fulu blobs

Open
#23,909 4 comments 1 reaction 1 assignee Claimed by @lystopad View on GitHub
Caplin
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
465

Description

### System information
Erigon version: `erigon version 3.6.1-0c4d9c91`

OS & Version: `Linux`

Erigon Command (with flags/config): /opt/ethmain/core/erigon --chain=mainnet --networkid=1 --datadir=/mnt/ethmain/node --prune.mode=archive --torrent.download.rate=64mb --http --http.addr=0.0.0.0 --http.port=8545 --http.vhosts=* --http.api=admin,eth,net,web3,debug,trace,txpool,erigon --trace.maxtraces=2000 --ws --ws.port=8546 --rpc.batch.limit=10000 --rpc.returndata.limit=1000000 --rpc.gascap=0 --rpc.txfeecap=0 --beacon.api.addr=0.0.0.0 --be acon.api.port=3500 --beacon.api.cors.allow-origins=* --beacon.api=beacon,builder,config,debug,events,node,validator,lighthouse --port=30303 --nodekey=/mnt/ethmain/node/nodekey --caplin.blobs-immediate-backfill --caplin.states-archive --ca plin.blocks-archive --caplin.blobs-archive --caplin.blobs-no-pruning

Consensus Layer: `Caplin`

Consensus Layer Command (with flags/config): No

Chain/Network: `1`

### Summary

My node follows the chain head normally, but historical Blob Sidecars are missing. After upgrading to v3.6.1, BlobHistoryDownloader is running, but its backward scan advances at only approximately 0.03–0.04 slots/s.

Could you help confirm whether this behavior is expected and how to accelerate historical P2P blob backfilling?

### Backfill logs
```bash
[INFO] [09-10|10:43:34.236] [BlobHistoryDownloader] Downloading blobs backwards slot=15181681 to=8626176 slots/sec=0.03 progress=0.0% eta=54631h45m8s
[INFO] [09-10|10:47:04.246] [BlobHistoryDownloader] Downloading blobs backwards slot=15181673 to=8626176 slots/sec=0.04 progress=0.0% eta=47802h44m46s
[INFO] [09-10|10:50:04.256] [BlobHistoryDownloader] Downloading blobs backwards slot=15181665 to=8626176 slots/sec=0.04 progress=0.0% eta=40974h2m55s
```

### Snapshot and hot-storage coverage

The snapshot directory contains 605 blobsidecars.seg files. Their filename ranges are contiguous:
[8,620,000, 14,670,000)

This is a filename-range check, not a verification of snapshot contents or loaded indexes.
The hot-storage directory is:
```bash
/mnt/ethmain/node/caplin/blobs/
```

It contains numeric buckets 1474 through 1518. Buckets 1467 through 1473 are absent, suggesting a potential gap after the snapshot boundary.

However, existing buckets also appear to have missing data; directory presence does not imply complete coverage.

### Confirmed API discrepancy
On this same node, slot 14740001 has 9 KZG commitments:

```bash
curl -s http://127.0.0.1:3500/eth/v2/beacon/blocks/14740001 |
jq '{
version,
slot: .data.message.slot,
commitments: (.data.message.body.blob_kzg_commitments | length),
finalized,
execution_optimistic
}'
```

Response:

```bash
{
"version": "fulu",
"slot": "14740001",
"commitments": 9,
"finalized": true,
"execution_optimistic": false
}
```

But the Blob Sidecar endpoint returns an empty array:

```bash
curl -s http://127.0.0.1:3500/eth/v1/beacon/blob_sidecars/14740001
{
"data": [],
"execution_optimistic": false,
"finalized": true,
"version": "fulu"
}
```

### Questions
Is this backfill speed expected for Fulu blocks? Could repeated PeerDAS column-recovery timeouts explain it, and which logs or metrics would confirm that?
Is there a supported way to accelerate backfilling—for example, recovery concurrency, peer selection, traffic limits, or connections to peers retaining historical columns?
Would enabling --caplin.snapgen help in this situation, or would snapshot generation remain blocked until missing sidecars are recovered? Should I avoid restarting while the current backward scan is in progress?

I would appreciate guidance on whether the node is behaving as expected and the recommended next steps to restore complete historical Blob availability.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.