erigontech / erigontech/erigon

cl/antiquary: a missing blob sidecar wedges blob antiquation permanently, retried every 12s

Open
#23,224 3 comments 0 reactions 1 assignee Claimed by @lystopad View on GitHub
Caplin
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

### Symptom

A Gnosis archive node (`release/3.5` at `ef999bab93`) repeated this at a flat 12-second period for 65 minutes, until the process was OOM-killed:

```
[INFO] [08-12|13:23:18.835] [Antiquary] Antiquating blobs app=caplin from=29400000 to=29479999
[EROR] [08-12|13:23:20.318] [Antiquary] Failed to antiquate blobs err="blob storage count mismatch at slot 29403906: 1 != 0"
```

The beacon block at slot 29403906 carries one KZG commitment; `blobStorage.KzgCommitmentsCount` for its root returns zero.

### Affected versions

Not release-specific. `release/3.4`, `release/3.5`, `release/3.6` and `main` all carry the same fatal `blob storage count mismatch` check in `DumpBlobSidecarsRange`, the same two sibling fatal paths (`blob sidecars count mismatch`, `blob sidecars not found`), and the same `retirementTicker := time.NewTicker(12 * time.Second)` with a log-and-continue error handler.

What differs is the consequence. With #23223 fixed (merged to `main` and `release/3.6`), the wedge no longer leaks compressor workers, so it stops being fatal — it degrades to burning CPU on ~3900 beacon-body reads every 12 seconds and blocking blob pruning indefinitely. On a branch without that fix it is fatal within about an hour.

### Why it never recovers

`DumpBlobSidecarsRange` (`db/snapshotsync/freezeblocks/caplin_snapshots.go`) treats the mismatch as fatal, and `antiquateBlobs` is driven by a 12-second ticker that only logs the error (`cl/antiquary/antiquary.go`). Nothing about the node's state changes between attempts, so:

* blob snapshots never advance past 29400000 — `FrozenBlobs()` is frozen;
* blobs are therefore never pruned from `chaindata` by the post-dump prune step, so the blob tables grow without bound;
* roughly 3900 beacon block bodies plus blob-storage lookups are re-read every 12 seconds, indefinitely;
* each attempt abandons a `seg.Compressor` before `Compress()`, which leaks its pattern workers — #23223. That leak is what actually OOM-kills the node; this issue is the condition that drives it.

### Open question

What should a genuinely missing sidecar do? Options: backfill it from peers before dumping; record the gap and continue; or fail once and back off instead of retrying a range that cannot succeed. Retrying every 12 seconds forever is wrong under any of them.

Worth deciding whether a missing sidecar is legitimate rather than corruption. `pruneBlobDistance` is 128600 slots unless `--caplin.blobs-archive` or `--caplin.blob-pruning-disabled` is set (`cmd/caplin/caplin1/run.go:286-289`), and `BlobStore.Prune` runs from `cl/phase1/stages/cleanup_and_pruning.go` on that retention window, independently of antiquation progress. On Gnosis 128600 slots is about 7.4 days. With the tip near 29480000 the retention floor sits around 29351400 — still below 29400000, so retention does not explain this particular gap yet, but it leaves only ~2.8 days of margin before the pruner starts consuming slots the wedged antiquary has not dumped. If that is reachable, the sanity check is wrong to be fatal and the two components need an ordering guarantee.

I have not confirmed which of these produced slot 29403906 on this node. One datum that may narrow it: the error appears in a single log archive, starting 12:17:55 on 2026-08-12 — so antiquation had been advancing normally and only wedged once `FrozenBlobs()` reached 29400000.

### Workaround

`--caplin.snapgen=false` stops the loop (and with it the leak in #23223), at the cost of no caplin snapshot generation. The un-prunable blobs remain.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.