filecoin-project / filecoin-project/boost

Support Data Segment Index

Open
#1,258 2 comments 0 reactions 1 assignee Claimed by @willscott View on GitHub
kind/enhancement need/triage
Dominant language
Go
Stars
121
Forks
79
Avg merge
10d 14h
Merged PRs (30d)
1

Description

### Boost component

- [X] boost daemon - storage providers

### Improvement Suggestion

Currently boost indexes pieces using go-car to generate a local index of CIDs contained in the piece for use in retrievals and to notify network indexers.

This format has been extended recently, per the [go-data-segment](https://github.com/filecoin-project/go-data-segment) library. This wrapper can be used to parse pieces, in particular by parsing a trailing index if it exists.

as @kubuxu notes:
> Here is the code for parsing the data segment index: https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/parse_index.go
After the index is parsed, you should use IndexData#ValidEntries()` to gather valid data segments on which you can access information of each segment using PieceCID() UnpaddedOffset() UnpaddedLength(): https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/index.go#L156-L173

This is currently managed by the [dagstore](https://github.com/filecoin-project/dagstore/blob/master/dagstore_async.go#L123)
I think the lid variant will be [this](https://github.com/filecoin-project/boost/blob/main/extern/boostd-data/ldb/service.go#L194)? (can you confirm, @nonsense?)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.