libp2p / libp2p/go-libp2p-pubsub

`Many*` methods: `NextMany`, `ValidateMany`, `PublishMany`

Open
#644 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
364
Forks
221
PR merge metrics
No merged PRs in 30d

Description

For pulsatile, high-volume broadcast workloads, we can get mechanistic gains simply by treating these as batches end-to-end. This should be built as an opt-in behaviour and the application must be ready to handle it. That is, we should prefer dedicated APIs over magically attempting to batch under the hood (the tradeoff space is similar to Nagle / TCP_NO_DELAY).

The proposal is to implement a set of `*Many` methods and interfaces that applications consume directly for the topics where this behaviour makes sense. (Like Ethereum attestations).

**Forwarding path**
- Consume what's available in the ingress queue at once by passing in a buffer to avoid reallocs (`NextMany(ctx context.Context, buffer []*Message) (uint, error)`).
- Validate everything in batch (if the validator implements a `ValidatorMany([]*Message) ([]bool, error)` interface).
- Forward all at once, chunking in MTU-sized RPCs (reconciling with `IDONTWANTs` from peers, i.e. dropping the attestations they don't want).

**Publishing path**
- Introduce similar mechanistic optimizations to the existing `PublishBatch`. In Ethereum, this would be invoked by a beacon node sending many attestations at once (e.g. many validators attesting on the same slot, no matter the committee because the topic is part of Message).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating the existing Next, validator, and PublishBatch entry points, then trace the forwarding and publishing paths. Done should mean the proposed opt-in Many APIs and interfaces are designed and implemented consistently across those paths, including the stated batching and chunking behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.