[ipfs/go-bitswap] Failure to fetch content from connected nodes
- Dominant language
- Go
- Stars
- 316
- Forks
- 163
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 8
Description
### Environment
IPFS Version:
```
go-ipfs version: 0.9.1-dc2715a
Repo version: 11
System version: amd64/linux
Golang version: go1.15.2
```
We are running a private IPFS cluster with 2 or more nodes (docker/k8s)
### Description
As an example, we've got a network with 2 nodes - `node1` and `node2`. After startup, we observe that `node2` cannot fetch content from `node1` (a bootstrap peer) because it is not registered as a `Partner`. We see on `node1` that `node2` is registered as a `Peer/Partner`. I tried adding some additonal debug logs and best I could tell, the steps to register a `BitSwapNetwork Receiver` to receive events from the network was not complete before `node2` received a "connected" notification from `node1`. It seems to me like this is a race condition and we see this occur 90% of the time, when we spin up nodes (ref: https://github.com/ipfs/go-bitswap/blob/0e81f7c1fef795a8d34cdd494fa2d8fdc045d199/bitswap.go#L215 ). As a throw of dice, I tried to move up the call _before_ any options/decision engine is setup, this did not make a whole lot of difference to the outcome.
### Workaround
* Execute `ipfs swarm peer disconnect
`* Execute `ipfs swarm peer connect `
We can see that the [PeerConnected](https://github.com/ipfs/go-bitswap/blob/0e81f7c1fef795a8d34cdd494fa2d8fdc045d199/bitswap.go#L528) handler is invoked from [Connected](https://github.com/ipfs/go-bitswap/blob/0e81f7c1fef795a8d34cdd494fa2d8fdc045d199/network/ipfs_impl.go#L428), which adds the Peer to the peer list. After this, `WANT_HAVE` is sent as expected to the peer and content is retrieved.
PS: Originally, I created the following [issue](https://github.com/ipfs/go-ipfs/issues/8346) in `go-ipfs`, but after further investigation, it feels like `go-bitswap` is the right place to report this.
I'd love to hear any suggestions/thoughts for how this can be addressed since I have a limited view of the architecture/intent for how connections are supposed to work. I have attached logs and output of commands from both nodes.
[ipfs-config-node1.txt](https://github.com/ipfs/go-bitswap/files/7079480/ipfs-config-node1.txt)
[ipfs-config-node2.txt](https://github.com/ipfs/go-bitswap/files/7079481/ipfs-config-node2.txt)
[ipfs-id-node1.txt](https://github.com/ipfs/go-bitswap/files/7079482/ipfs-id-node1.txt)
[ipfs-id-node2.txt](https://github.com/ipfs/go-bitswap/files/7079483/ipfs-id-node2.txt)
[ipfs-node1.log](https://github.com/ipfs/go-bitswap/files/7079484/ipfs-node1.log)
[ipfs-node2.log](https://github.com/ipfs/go-bitswap/files/7079485/ipfs-node2.log)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.