FilOzone / FilOzone/dealbot

Retrieval pre-check reports pieceLive=false ~40s after the piece is added on chain, marking healthy deals cleaned_up

Open
#686 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3
Forks
9
Avg merge
4d 14h
Merged PRs (30d)
6

Description

`retrieval_skipped_piece_missing` is firing on mainnet against pieces that were added on chain seconds earlier and have never been scheduled for removal.

One deal, dealbot-backend v1.12.0, 2026-08-20 UTC:

```
14:21:29.454 pieces_added pod vgmcl
14:22:02.256 pieces_confirmed pieceIds:[857] pod vgmcl
14:22:03.998 deal_creation_completed pod vgmcl
14:22:09.497 retrieval_skipped_piece_missing pod sscln pieceId: 857
"PDP pieceLive=false; marked deal cleaned_up and skipped retrieval"
```

Piece 857 was added on chain in block 6297643, timestamp 14:21:30. 5.5 seconds after the deal is logged complete, a different pod in the same deployment reports it not live and writes the deal off. A second deal the same day is identical in shape: `pieces_confirmed [861]` 11:23:02.971, `deal_creation_completed` 11:23:06.006, skipped 11:23:11.437.

## Scale

Every `retrieval_skipped_piece_missing` on mainnet from 2026-08-13 15:00 to 2026-08-20 15:00 UTC, cross-checked against indexed `fwss_piece_added` and `pdp_pieces_removed` records. 232 events across 232 distinct deals:

- 232 of 232 pieces are present on chain
- 0 appear in any removal event
- gap from the block timestamp of the add to the skip: 34s min, 70s max, 41s mean
- 0 outside that window, 0 skipped before their own add

| provider | skips |
|---|---|
| 1 ezpdpz-main | 82 |
| 36 Cello | 81 |
| 2 beck-main | 47 |
| 30 foc-mainnet | 10 |
| 32 ezpdpz-main2 | 10 |
| 34 StorageUsgp2 | 2 |

Both example pieces return `pieceLive = true` now. For 774/857 an `eth_call` to `PDPVerifier.pieceLive(774, 857)` returns false at block 6297642 and true at block 6297643, the block the add landed in.

## Effect

The deal is marked `cleaned_up` and the retrieval candidate query filters on `cleaned_up = false`, so it leaves the sampling pool. No code path sets `cleanedUp` back to false once set. That is 232 deals in the week above.

The provider dashboard's `Retrieval success%` column excludes only `pending`, so `skipped.piece_missing` counts as a non-success. At 2026-08-20 16:55 UTC ezpdpz-main reads 403 successes and 29 skips over 72h with no retrieval failures recorded in the window, which puts the column below the 97% approval threshold on a check that never failed.

The pre-check was added in #561.

Happy to pull more samples or run anything specific.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by tracing the retrieval pre-check introduced in #561 and the code that emits retrieval_skipped_piece_missing and marks deals cleaned_up. Use the reported piece-added, confirmation, and skip timings to reproduce the consistency window; done means newly added on-chain pieces are not incorrectly excluded from retrieval or permanently marked cleaned_up.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, blockchain
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.