ChainSafe / ChainSafe/gossamer

PeerState connection slot is not released on network error

Open
#3,758 0 comments 0 reactions 1 assignee Claimed by @P1sar View on GitHub
A-debug P-low S-network T-bug
Dominant language
Go
Stars
454
Forks
144
PR merge metrics
No merged PRs in 30d

Description

## Describe the bug

When node is running it writes all outgoing and incoming connection in to the slots that has limited capacity. But when libp2phost.connect happens with error there is no fallback route for us to free occupied connection slot inside PeerState.
Error happens pereodically [here](https://github.com/ChainSafe/gossamer/blob/development/dot/network/service.go#L718).
Another problem is that apprently before libp2p ibrary version 32.2 there is a bug that does not bubbleups an error so this happens unnoticeably this statement also requires verification.

Mentioned below log was done within this [branch](https://github.com/ChainSafe/gossamer/pull/3755) with additional info logs added
## Expected Behavior
IF libp2p error happens we should free occupied perstate slot.
[here](https://github.com/ChainSafe/gossamer/blob/development/dot/network/service.go#L718)

-

## Current Behavior
PerrState slot is not getting released when libp2p.connect error hapens
```
failed to negotiate security protocol: error reading handshake message: noise: message is too short
```

-

## Possible Solution
Basically handshake error is out of scope of this issue and generally we just need to add communication between network.service and peerstate so we can free occupied slot in case of error.

-

## To Reproduce
Steps to reproduce the behaviour:
Run integration test
1. MODE=rpc go test ./tests/rpc/system_integration_test.go -v
2. libp2p version should be 32.2 before that error happens silently without bubbling up
3.

## Log output

Log Output
```
system_integration_test.go:54: Node alice-0 is ready
alice-0 2024-02-13T20:32:07+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:07+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:09+01:00 INFO peer count 0 pkg=network
alice-0 2024-02-13T20:32:09+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:09+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:10+01:00 INFO HANDLE PEER FOUND BY MDNS {12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: [/ip4/127.0.0.1/tcp/7005 /ip4/192.168.1.133/tcp/7005]} pkg=network
2024-02-13T20:32:10+01:00 INFO PESRSET ADD PEER 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Inserting peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 0 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO tryOutgoing outgoing slot increase for 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO Sent connect message to peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7 pkg=peerset
2024-02-13T20:32:10+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
2024-02-13T20:32:10+01:00 INFO PROCESS NEW MESSAGE {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network
2024-02-13T20:32:10+01:00 INFO PROCESS MESSAGE CONNECT {Status:0 setID:0 PeerID:12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7} pkg=network
alice-0 2024-02-13T20:32:10+01:00 WARN failed to open connection for peer 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: failed to dial: failed to dial 12D3KooWHzKQ6E5DSDXVAUvPT4vruuXBc4zA44srfB7gPrHyqGS7: all dials failed
* [/ip4/127.0.0.1/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short
* [/ip4/192.168.1.133/tcp/7005] failed to negotiate security protocol: error reading handshake message: noise: message is too short pkg=network
alice-0 2024-02-13T20:32:10+01:00 INFO built block 1 with hash 0xdf9ea7f6c66d5ffc3193087cfd6477cc53b26f7443af48eb2a40bdb6ce6f1bce, state root 0xec9b9073553a22b406a9d390768b484767f75eb1f762a335bd0f8008f54e34fd, epoch 0 and slot 284642121 pkg=babe
alice-0 2024-02-13T20:32:10+01:00 INFO 🔨 finalised block #0 (0x276bfa91f70859348285599321ea96afd3ae681f0be47d36196bac8075ea32e8), round 1, set id 0 pkg=state
system_integration_test.go:54: Node bob-1 is ready
alice-0 2024-02-13T20:32:11+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
2024-02-13T20:32:11+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
alice-0 2024-02-13T20:32:12+01:00 INFO peer count 0 pkg=network
alice-0 2024-02-13T20:32:13+01:00 INFO Allocating slots for 0 pkg=peerset
2024-02-13T20:32:13+01:00 INFO hasFreeOutgoingSlot 0, numout 1 maxout 10 pkg=peerset
```


## Specification

- go version: 1.22
- gossamer version:
- gossamer commit tag:
- gossamer commit hash:
- operating system:
- additional links:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.