Discussion: is one graphsync.ResponseProgress message per Node in a graph reasonable?
- Dominant language
- Go
- Stars
- 103
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
A `Request()` returns a channel that gives you all of the _Nodes_ of a DAG traversal, rather than the _Blocks_ of that traversal.
In datatransfer, I see that it’s just doing a `for range req.responseChan {}` so they’re not used, I don’t know where else they might be consumed though.
But for a DAG with very complex blocks (like the filecoin chain, for example), this means that if someone uses a matching selector, they could trigger the creation of a `graphsync.ResponseProgress` for every single point of data (Node) within each of the blocks.
So, for example, running the plain blockchain used in the test cases, with 99 blocks, each of which is just `{Parents:[CID],Messages:[bytes,...]}`. Even with this very simple block format and a selector that limits itself to exploring `Parents`, each block generates two progress messages, one for visiting the `Parents` and one for visiting the first element of the list. So a 99 block chain, with this super-simple structure, and a selector that’s limiting itself to just `Parents` generates 198 messages.
Since we use `WalkAdv` it doesn’t even matter if it’s a matching or explore selector, it’ll trigger this for every single data element within a block.
Which goes to some of the concerns I’ve been raising recently about how we use selectors practically and the distinction between Nodes and Blocks getting in the way—what would the user of this API expect from a `Request()`? Is it reasonable that they get an exhaustive list of messages for each Node, or would they more logically expect one for each Block? Or maybe neither since we’re really just using it for a block store. And in terms of performance, what are the implications of having so many messages per block. Even for DAG-PB blocks, we’re going to be triggering over 100 messages for a single block in many cases (e.g. MFS or just dense directory blocks).
Contributor guide
Assessment
This issue has not been assessed yet.