erigontech / erigontech/erigon
db/snapshotsync: RemoveOverlaps cannot unlink caplin beacon-block overlaps, leaving them orphaned
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
`BaseRoSnapshots.RemoveOverlaps` finds its work with a raw directory scan (`snaptype.Segments(s.dir)`, `db/snapshotsync/snapshots.go:1560`), but its only physical effect is `retireSegmentsNotInList(RetireReasonMerged, keepNames)` (`:1609`), which walks `s.dirty`. A file absent from `s.dirty` can never be unlinked.
For `CaplinSnapshots`, every path that populates `dirty` goes through `OpenFolder` -> `SegmentsCaplin` (`:2049`), which applies `NoGaps(NoOverlaps(l))` to beaconblocks. Those two filters drop exactly the files `findOverlaps` later targets:
- `NoOverlaps` (`:59`) collapses entries sharing a `From` and keeps the last, i.e. the widest
- `NoGaps` (`:87`) drops any file with `to <= prevTo`, i.e. every contained subset
Measured by calling the real functions with a downloaded 100k file over ten local 10k dumps:
```
in: [{0 10000} {0 100000} {10000 20000} ... {90000 100000}]
after NoOverlaps: [{0 100000} {10000 20000} {20000 30000} ... {90000 100000}]
after NoGaps: [{0 100000}]
```
`OpenList` then detaches the dropped files with `RetireReasonWasDeletedFromDisk` (`:1321`), i.e. `canDelete=false` — close only, never unlink.
So a covered beaconblocks `.seg`/`.idx` stays on disk permanently, and every later `RemoveOverlaps` re-discovers it from the disk scan and re-skips it. Blob sidecars are unaffected: `SegmentsCaplin` appends them unfiltered (`:2063-2065`).
Effect: beaconblocks disk usage never shrinks after a merge-limit change, or on a datadir mixing downloaded and locally dumped ranges.
It also blocks #23412 for this family. A node-path `RemoveOverlaps` call for caplin blocks still reaches `onDelete` — that side is computed from the disk scan and the paths are correct, since `CaplinSnapshots.dir` is `dirs.Snap` — so the `.torrent` is unlinked and seeding stops for a `.seg` that then stays on disk. That is worse than not calling it at all, which is why #23412 wired only `CaplinStateSnapshots`.
Part of #23024, item 2.
Contributor guide
Research direction
Start in db/snapshotsync/snapshots.go, tracing BaseRoSnapshots.RemoveOverlaps, CaplinSnapshots.OpenFolder, SegmentsCaplin, and OpenList. Reproduce the overlap case with the described downloaded and local ranges, then verify that covered beaconblocks .seg and .idx files are removed while blob sidecars and retained segments remain intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100