Persistent peer identity during restart make it unable to publish message with gossipsub

Summary

When using QUIC transport exclusively with a persistent identity, the restart of the client of one node would make that node unable to publish message and getting an InsufficientPeers error. However, both swarm and gossipsub would report that there is a connected peer.

For example:

Both PeerA and PeerB has a static identity(and peer id derived from that identity) and connected to each other using QUIC transport exclusively, and gossipsub is able to send messages to each other.

Now PeerA restarts, connect to PeerB with the same identity, PeerA will be unable to publish any message and will always get InsufficientPeers, but the swarm.connected_peers() and gossipsub.all_peers() are returning PeerB as connected peer. PeerB can publish message to PeerA.

To reproduce this, we can simply modify the chat example:

First we add ed25519_dalek as a direct dependency, then add the following to the chat example:

// in header
use ed25519_dalek::{pkcs8::DecodePrivateKey, SigningKey};
...
// in main function:
let pem = std::fs::read_to_string("private_key.pem")?;
let mut verifying_key_bytes = SigningKey::from_pkcs8_pem(&*pem).unwrap().to_bytes();
let keypair = libp2p::identity::Keypair::ed25519_from_bytes(&mut verifying_key_bytes)?;
...
// use keypair as exisiting identity:
let mut swarm = libp2p::SwarmBuilder::with_existing_identity(keypair)
...
// comment out the tcp listening so we use quic exclusively:
//swarm.listen_on("/ip4/0.0.0.0/tcp/0".parse()?)?;

Then change the private_key.pem for PeerA and PeerB, we can use openssl to generate the pem:

openssl genpkey -algorithm Ed25519 -out private_key.pem

Make sure PeerA and PeerB is using different private key, and restart one of the client, you will see Publish error: InsufficientPeers on the restarted node.

Expected behavior

The peer should be able to publish message even after restart

Actual behavior

The peer restarted will report InsufficientPeers, the peer not restarted is able to publish message to the peer restarted.

Relevant log output

Enter messages via STDIN and they will be sent to connected peers using Gossipsub
Local node is listening on /ip4/127.0.0.1/udp/62392/quic-v1
Local node is listening on /ip4/192.168.125.145/udp/62392/quic-v1
mDNS discovered a new peer: 12D3KooWC1tB5J6W7NWwzXVFHZBggNLU1T5etuTtmpnf9WBQaJCm
mDNS discovered a new peer: 12D3KooWC1tB5J6W7NWwzXVFHZBggNLU1T5etuTtmpnf9WBQaJCm
test
Publish error: InsufficientPeers

Possible Solution

Current workarounds including:

• Use TCP based transports as fallback.
• Don't reboot too fast. Shut down the node and wait for ~10 seconds until the remote node closed the connection, which can be detected with an swarm::ConnectionError::IO event, then start the node. Starting the node after remote closed the connection will not trigger InsufficientPeers

Version

I tried "0.53.2" as well as running the chat example directly on master with 3e57cf494792031459d7f831408af24dbedc1a0b.

Would you like to work on fixing this bug ?

Yes

Start with the chat example and its SwarmBuilder::with_existing_identity setup, then reproduce the QUIC-only restart scenario using the provided Ed25519 key steps. Compare the swarm and gossipsub peer state around restart and the InsufficientPeers publish error. Done means the restarted peer can publish messages to its connected peer without that error.