erigontech / erigontech/erigon

seg integrity: failFast=false reports success and exits 0 after logging integrity errors

Open
#23,611 0 comments 0 reactions 1 assignee Claimed by @sudeepdino008 View on GitHub
type:bug
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

With `--failFast=false`, integrity checks log every mismatch, return `nil`, print `success`, and exit 0. A wrapper script, a CI job, or an operator reading the last line will conclude the datadir is clean.

Hit on a gnosis archive datadir with a genuine RCache hole: ~15,000 blocks where receipts are missing, hundreds of `EROR` lines, and then:

```
EROR[08-26|16:04:36.613] integrity error: check-rcache-root-at-blk: receipt root mismatch at block 46844899: computed=0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, header=0xfc07657bc752f7ae73bbcdafbd458f4e0643ff394d467dc0bf42c27ae01fedb0
INFO[08-26|16:04:36.701] [check-rcache-root-at-blk-range] success
```

`computed` is `empty.ReceiptsHash`, so the check derived a root over zero receipts — a real defect, reported as success.

## Where it happens

The mismatch is logged and discarded, so the chunk returns `nil`:

```go
// db/integrity/rcache_receipt_root.go:152-159
if computedRoot != header.ReceiptHash {
mismatch := fmt.Errorf("%w: ... receipt root mismatch at block %d ...", ErrIntegrity, blockNum)
if failFast {
return mismatch
}
log.Error(mismatch.Error()) // <- dropped here
}
```

`nil` then propagates unchanged through `parallelChunkCheck` (`db/integrity/rcache_no_duplicates.go:179-184`), the check wrapper logs `ReceiptRootIntegrity: done err=nil` (`rcache_receipt_root.go:41`), and the CLI action prints `success` (`cmd/utils/app/snapshots_cmd.go:497-500`). The `seg integrity` runner is affected the same way: `runCheck` returns `nil`, so it logs `[integrity] done check=ReceiptRootIntegrity` and the whole run exits 0.

This is not specific to the receipt check — `grep -n "if failFast" db/integrity/*.go` shows the same log-and-drop shape across `commitment_integrity.go`, `no_gaps_in_canonical_headers.go`, `integrity_kvi.go` and `caplin_state_integrity.go`, so the fix should be applied consistently rather than only where it was noticed.

## Suggested fix

The repo already has the pattern this needs, for torrent verification (`snapshots_cmd.go:1570-1575`): keep going, record the failure, join it into the returned error via a deferred hook.

1. Count mismatches per check instead of dropping them, and return an error when the count is non-zero — including when `failFast=false`. `failFast` should control *when the run stops*, not *whether failures are reported*.
2. Log a summary per check (`mismatches=N`), so the scale is visible without counting `EROR` lines by hand.
3. Have the CLI actions print `success` only on a genuinely clean result.

Two related reporting problems in the same area, worth folding in:

- **Progress percentage is unreachable under sampling.** `parallelChunkCheck` computes `progress` against `totalChunks` over the entire block range while only the sampled chunks are ever run (`rcache_no_duplicates.go:152,165`), so at `--sample=0.01` it climbs to ~1% and then the check finishes. Observed: `progress=0.8%` immediately followed by `done`.
- **An empty stream is indistinguishable from empty blocks.** In `checkRCacheRootAtBlkChunk`, if `ReceiptCacheV2Stream` yields nothing, the tail loop (`rcache_receipt_root.go:205-209`) verifies every block against an empty receipt list, so a failed or empty read is reported as N independent root mismatches rather than one "no receipts found for txNum range [x,y]" error. An explicit guard would have pointed straight at the cause instead of requiring a second run at `--sample=1.0` to work out what was going on.

## Reproducing

Any datadir with a receipt-root mismatch will do:

```bash
./build/bin/erigon seg check-rcache-root-at-blk-range --datadir= \
--from= --to= --sample=1.0 --failFast=false
echo $? # 0, after logging every mismatch
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.