Change UnixFS dag-pb protobuf field ordering
- Dominant language
- HTML
- Stars
- 1.2k
- Forks
- 247
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 4
Description
The `.proto` definition of a `PBNode` is this ([spec](https://ipld.io/specs/codecs/dag-pb/spec/#serial-format)):
```proto
message PBNode {
// refs to other objects
repeated PBLink Links = 2;
// opaque user data
optional bytes Data = 1;
}
```
Implementations are [expected](https://github.com/ipfs/specs/pull/531) to write the repeated `Links` messages to the output buffer first, then the `Data` field, even though the field IDs are ordered the opposite way (not disallowed by the protobuf spec, though some off the shelf encoders will write in ID order).
When the `PBNode` represents a directory, the `Links` objects could be either flat directory entries, or HAMT shard entries - the information needed to ascertain this is contained in the `Data` field.
This means when processing an incoming `PBNode` message, we typically read all of the `Links`, and then use the `Data` to decide how to process them.
If we are performing a graph traversal, a streaming parser would let us process the `PBNode` message as it arrives, and to select the `Link` message we wish to traverse through or resolve to, however this is not currently possible since we need to process the `Data` field before we can return a `Link`. This can add significant overhead when there are many thousands of `Link`s.
We should allow ordering the `Data` field first, this would enable the streaming use-case and help with IPFS code running in resource-constrained environments such as web browsers.
Since [IPIP-499](https://github.com/ipfs/specs/pull/499) we now have CID profiles which could be an upgrade path for the network. This proposal could be part of a `unixfs-v1-2026` profile.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.