libp2p / libp2p/go-libp2p-pubsub
`Many*` methods: `NextMany`, `ValidateMany`, `PublishMany`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 364
- Forks
- 221
- PR merge metrics
- No merged PRs in 30d
Description
For pulsatile, high-volume broadcast workloads, we can get mechanistic gains simply by treating these as batches end-to-end. This should be built as an opt-in behaviour and the application must be ready to handle it. That is, we should prefer dedicated APIs over magically attempting to batch under the hood (the tradeoff space is similar to Nagle / TCP_NO_DELAY).
The proposal is to implement a set of `*Many` methods and interfaces that applications consume directly for the topics where this behaviour makes sense. (Like Ethereum attestations).
**Forwarding path**
- Consume what's available in the ingress queue at once by passing in a buffer to avoid reallocs (`NextMany(ctx context.Context, buffer []*Message) (uint, error)`).
- Validate everything in batch (if the validator implements a `ValidatorMany([]*Message) ([]bool, error)` interface).
- Forward all at once, chunking in MTU-sized RPCs (reconciling with `IDONTWANTs` from peers, i.e. dropping the attestations they don't want).
**Publishing path**
- Introduce similar mechanistic optimizations to the existing `PublishBatch`. In Ethereum, this would be invoked by a beacon node sending many attestations at once (e.g. many validators attesting on the same slot, no matter the committee because the topic is part of Message).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by locating the existing Next, validator, and PublishBatch entry points, then trace the forwarding and publishing paths. Done should mean the proposed opt-in Many APIs and interfaces are designed and implemented consistently across those paths, including the stated batching and chunking behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100