Backpressure between components

What is backpressure

A slow consumer should slow down (i.e. backpressure) a fast producer.

Why do we need backpressure

• Prevent unbounded growth of buffers.
• Potential DOS defense.
• Potential lower latencies.
• Potential better resource allocation.

See also coding guidelines - Bound everything.

Where do we enforce backpressure

• User -> Swarm

  • No backpressure
  • Useful when dialing a lot of peers via Swarm::dial.
  • ConnectionLimit enforces boundedness at least in the Pool.
  • Might not be worth fixing, i.e. bursts might be fine.

• User -> NetworkBehaviour

  • No backpressure
  • User can access NetworkBehaviour via Swarm::behaviour_mut
  • Useful e.g. when doing large amounts of DHT lookups via libp2p-kad, many libp2p-request-response requests, ...
  • Potential solution
    • In the case of libp2p-kad have a Kademlia::poll_find_closest_ready that needs to be polled before Kademlia::find_closest similar to Sink::poll_ready.

• Swarm -> NetworkBehaviour

  • No backpressure
  • Only system notification events (e.g. connection established), thus not important.
  • Inbound connection
    • Can only be droppped
    • We have static limits via Swarm
    • In the future, with generic connection management, this can be dynamic, e.g. based on your system memory https://github.com/libp2p/rust-libp2p/issues/2824

• NetworkBehaviour -> ConnectionHandler

  • No backpressure
  • Swarm does not poll NetworkBehaviour in case it could not deliver previously returned event from the NetworkBehaviour to the destined ConnectionHandler.
    https://github.com/libp2p/rust-libp2p/blob/f9b4af3d9d5b12b20756e496349b0866baa862da/swarm/src/lib.rs#L1032-L1034
  • NetworkBehaviour is blocked on single slow ConnectionHandler
  • Potential solutions
    • Drop the event
      • NetworkBehaviour already needs to handle the case where the connection closes and thus the event is never delivered to the ConnectionHandler.
      • When sending two events, first might be dropped (ConnectionHandler busy) while the second might be delivered. Not intuitive.
    • Return event back to the NetworkBehaviour
    • Add NetworkBehaviour::poll_connection_handler_event, providing a list of ConnectionIds with ConnectionHandlers that are ready to receive another event.

• NetworkBehaviour -> Swarm

  • NetworkBehaviourAction::Dial

• ConnectionHandler -> NetworkBehaviour

  • Backpressure
  • ConnectionHandler is blocked on slow NetworkBehaviour
  • Expected behaviour

• Connection -> ConnectionHandler

  • ConnectionHandler::inject_fully_negotiated_inbound
    • Inbound can only drop
    • how about poll_inbound_ready
    • Note that multistream-select needs to run first, otherwise we can't map to a specific ConnectionHandler
  • ConnectionHandler::inject_fully_negotiated_outbound
    • how about poll_outbound_substream
    • Or the ConnectionHandler needs to internally enforce a limit of pending outbound connections. (Lots of duplication given the amount of ConnectionHandlers)
  • ConnectionHandler::inject_event
    • No backpressure
    • Possible solution
      • Add ConnectionHandler::poll_inject_event_ready

• ConnectionHandler -> Connection

  • ConnectionHandlerEvent::OutboundSubstreamRequest
    • Can only be dropped

• Stream

  • Creation of outbound streams
    • Backpressure is only enforced by QUIC
    • Potentially one day by Yamux (https://github.com/libp2p/rust-yamux/issues/150)
    • Potentially one day with https://github.com/libp2p/specs/pull/394
    • For now, we need to drop streams
  • Bytes on stream
    • Fine with Yamux, QUIC and potentially one day qmux
    • No backpressure on streams with mplex https://github.com/libp2p/specs/pull/402

Related resources

• https://github.com/libp2p/rust-libp2p/pull/1602
• https://github.com/libp2p/rust-libp2p/issues/3041
• https://github.com/libp2p/rust-libp2p/blob/master/docs/coding-guidelines.md

Start with the backpressure checklist and the referenced swarm/src/lib.rs section around lines 1032-1034, then review the linked coding guidelines and related issues or pull requests. The issue lists several possible solutions across Swarm, NetworkBehaviour, ConnectionHandler, and streams, but does not define one implementation scope or a concrete done condition.