Research libp2p DDoS attack mitigation

Problem to be solved

Charon nodes in a cluster communicate via libp2p. There are multiple libp2p protocols inside charon (dkg/qbft/ping/parsigex). A Byzantine node can DDoS other nodes by spamming them with valid (or invalid) connections and messages, causing OOM and or CPU problems.

DDoS can happen on the following resources:

• Number of libp2p connections
• Number of messages per connection
• Size of messages

This is however a common problem, so there might be existing solutions to the problem.

Proposed solution

• Research different ways to mitigate DDoS attacks (by Byzantine cluster peers).
• Look at libp2p connmgr.ConnManager to limit connections
• Look at libp2p network.ResourceManager to limit memory, streams, connections, and file descriptors
• Look at something like for bandwith throttling: https://github.com/corverroos/rateconn
• Look at custom message rate limiters and custom message size rate limiters (both detecting and then throttling)

Write a document with findings and recommendations.

Out of Scope

Nothing needs to be implemented yet.

Start by reviewing libp2p's connmgr.ConnManager and network.ResourceManager, then compare rateconn and custom message and size limiters. Document the available mitigation approaches, their trade-offs for Charon's connections and protocols, and recommendations; no implementation is required.