Published messages can be dropped silently

Currently if the outbound message queues for all peers in a mesh/fanout are full, gossipsub drops the message silently, which has already bitten us various times (in stress testing, and during actual usage).

This happens here: https://github.com/libp2p/go-libp2p-pubsub/blob/master/gossipsub.go#L359

Proposal

• Extend the signature of Publish() so it can take functional options to configure the reliability characteristics we want.

  • WaitUntilQueued(One | All | float64) => blocks the publish until the message has been queued to as many peers as the arg indicates. Can be implemented easily.
  • WaitUntilSent(One | All | float64) => blocks the publish until the message has actually been dispatched to as many peers as the arg indicates. Requires deeper changes, but reliability is higher.

• Aggregate RPCs waiting in the queue. Messages are associative and can be merged, in theory.

• Make queue size configurable. It's currently hardcoded to 32, and cannot be changed.

Affected users

Filecoin, ETH2 gossipsub testing.

Start at gossipsub.go#L359 and trace how Publish handles outbound queues when all peer queues are full. Review the proposed Publish reliability options, dispatch behavior, and configurable queue size, then clarify which proposal is in scope before defining done as preventing silent drops with agreed reliability semantics.