Avoid duplicating the full piece buffer when parsing sampled-retrieval CARs
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 9
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
`readBlocks` in the sampled-retrieval piece validation wraps the incoming piece buffer in `new Uint8Array(pieceBytes)` before handing it to `CarReader.fromBytes`:
https://github.com/FilOzone/dealbot/blob/2452e7deeb177ff6e94383386458f2a914ebf679/apps/backend/src/retrieval-anon/piece-validation.service.ts#L192-L200
`pieceBytes` is a `Buffer`, which already is a `Uint8Array`, so that wrapper allocates a second full copy of the entire piece. Sampled pieces run up to ~500 MiB, so peak memory for one check is roughly 2x the piece size for no reason.
Fix: pass `pieceBytes` straight to `CarReader.fromBytes(pieceBytes)`. Same behavior, drops the redundant allocation.
This is pre-existing from #487 and untouched by the rename in #612, so tracking it on its own. The file moves to `apps/backend/src/sampled-retrieval/piece-validation.service.ts` once #612 lands.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in apps/backend/src/retrieval-anon/piece-validation.service.ts at readBlocks and inspect how the piece buffer is passed to CarReader.fromBytes. Make the allocation-free change described in the issue, then verify sampled-retrieval piece validation retains the same behavior and no longer duplicates the full buffer; account for the later path move to apps/backend/src/sampled-retrieval/piece-validation.service.ts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100