filecoin-project / filecoin-project/boost
Support Data Segment Index
- Dominant language
- Go
- Stars
- 121
- Forks
- 79
- Avg merge
- 10d 14h
- Merged PRs (30d)
- 1
Description
### Boost component
- [X] boost daemon - storage providers
### Improvement Suggestion
Currently boost indexes pieces using go-car to generate a local index of CIDs contained in the piece for use in retrievals and to notify network indexers.
This format has been extended recently, per the [go-data-segment](https://github.com/filecoin-project/go-data-segment) library. This wrapper can be used to parse pieces, in particular by parsing a trailing index if it exists.
as @kubuxu notes:
> Here is the code for parsing the data segment index: https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/parse_index.go
After the index is parsed, you should use IndexData#ValidEntries()` to gather valid data segments on which you can access information of each segment using PieceCID() UnpaddedOffset() UnpaddedLength(): https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/index.go#L156-L173
This is currently managed by the [dagstore](https://github.com/filecoin-project/dagstore/blob/master/dagstore_async.go#L123)
I think the lid variant will be [this](https://github.com/filecoin-project/boost/blob/main/extern/boostd-data/ldb/service.go#L194)? (can you confirm, @nonsense?)
Contributor guide
Assessment
This issue has not been assessed yet.