ipfs / ipfs/specs

Change UnixFS dag-pb protobuf field ordering

Open
#533 0 comments 2 reactions 0 assignees View on GitHub
need/triage
Dominant language
HTML
Stars
1.2k
Forks
247
Avg merge
5d 21h
Merged PRs (30d)
4

Description

The `.proto` definition of a `PBNode` is this ([spec](https://ipld.io/specs/codecs/dag-pb/spec/#serial-format)):

```proto
message PBNode {
// refs to other objects
repeated PBLink Links = 2;

// opaque user data
optional bytes Data = 1;
}
```

Implementations are [expected](https://github.com/ipfs/specs/pull/531) to write the repeated `Links` messages to the output buffer first, then the `Data` field, even though the field IDs are ordered the opposite way (not disallowed by the protobuf spec, though some off the shelf encoders will write in ID order).

When the `PBNode` represents a directory, the `Links` objects could be either flat directory entries, or HAMT shard entries - the information needed to ascertain this is contained in the `Data` field.

This means when processing an incoming `PBNode` message, we typically read all of the `Links`, and then use the `Data` to decide how to process them.

If we are performing a graph traversal, a streaming parser would let us process the `PBNode` message as it arrives, and to select the `Link` message we wish to traverse through or resolve to, however this is not currently possible since we need to process the `Data` field before we can return a `Link`. This can add significant overhead when there are many thousands of `Link`s.

We should allow ordering the `Data` field first, this would enable the streaming use-case and help with IPFS code running in resource-constrained environments such as web browsers.

Since [IPIP-499](https://github.com/ipfs/specs/pull/499) we now have CID profiles which could be an upgrade path for the network. This proposal could be part of a `unixfs-v1-2026` profile.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.