Retrieval pre-check reports pieceLive=false ~40s after the piece is added on chain, marking healthy deals cleaned_up
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 9
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
`retrieval_skipped_piece_missing` is firing on mainnet against pieces that were added on chain seconds earlier and have never been scheduled for removal.
One deal, dealbot-backend v1.12.0, 2026-08-20 UTC:
```
14:21:29.454 pieces_added pod vgmcl
14:22:02.256 pieces_confirmed pieceIds:[857] pod vgmcl
14:22:03.998 deal_creation_completed pod vgmcl
14:22:09.497 retrieval_skipped_piece_missing pod sscln pieceId: 857
"PDP pieceLive=false; marked deal cleaned_up and skipped retrieval"
```
Piece 857 was added on chain in block 6297643, timestamp 14:21:30. 5.5 seconds after the deal is logged complete, a different pod in the same deployment reports it not live and writes the deal off. A second deal the same day is identical in shape: `pieces_confirmed [861]` 11:23:02.971, `deal_creation_completed` 11:23:06.006, skipped 11:23:11.437.
## Scale
Every `retrieval_skipped_piece_missing` on mainnet from 2026-08-13 15:00 to 2026-08-20 15:00 UTC, cross-checked against indexed `fwss_piece_added` and `pdp_pieces_removed` records. 232 events across 232 distinct deals:
- 232 of 232 pieces are present on chain
- 0 appear in any removal event
- gap from the block timestamp of the add to the skip: 34s min, 70s max, 41s mean
- 0 outside that window, 0 skipped before their own add
| provider | skips |
|---|---|
| 1 ezpdpz-main | 82 |
| 36 Cello | 81 |
| 2 beck-main | 47 |
| 30 foc-mainnet | 10 |
| 32 ezpdpz-main2 | 10 |
| 34 StorageUsgp2 | 2 |
Both example pieces return `pieceLive = true` now. For 774/857 an `eth_call` to `PDPVerifier.pieceLive(774, 857)` returns false at block 6297642 and true at block 6297643, the block the add landed in.
## Effect
The deal is marked `cleaned_up` and the retrieval candidate query filters on `cleaned_up = false`, so it leaves the sampling pool. No code path sets `cleanedUp` back to false once set. That is 232 deals in the week above.
The provider dashboard's `Retrieval success%` column excludes only `pending`, so `skipped.piece_missing` counts as a non-success. At 2026-08-20 16:55 UTC ezpdpz-main reads 403 successes and 29 skips over 72h with no retrieval failures recorded in the window, which puts the column below the 97% approval threshold on a check that never failed.
The pre-check was added in #561.
Happy to pull more samples or run anything specific.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by tracing the retrieval pre-check introduced in #561 and the code that emits retrieval_skipped_piece_missing and marks deals cleaned_up. Use the reported piece-added, confirmation, and skip timings to reproduce the consistency window; done means newly added on-chain pieces are not incorrectly excluded from retrieval or permanently marked cleaned_up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, blockchain
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100