ipfs / ipfs/kubo

Goroutine count, Node Performance, and correlation with peer count

Open
#5,420 8 comments 0 reactions 0 assignees View on GitHub
kind/bug topic/perf
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

#### Version information:
```
go-ipfs version: 0.4.17-
Repo version: 7
System version: amd64/linux
Golang version: go1.10.3
```
Note that I'm also running an IPFS Cluster daemon on the same node which is connected to one other peer:
```
ipfs-cluster-service version 0.5.0
```

#### Type:
Possible goroutine leak or other bug

#### Description:

The number of goroutines running on my IPFS nodes appears to be highly correlated with the number of peers my node is connected to. In the past I would normally have between 600 -> 1000 peers, and I noticed overall "slow" performance to my node. One thing I noticed is that my nodes always had significantly high numbers of goroutines running constantly (10K+).

I suspected the poor performance was due to slow DHT querying with the large amount of peers so I lowered my peer count range (200 -> 500). Subsequently my node's performance was significantly better than before, and that my goroutine count was lower. Today while checking up on my monitoring system I noticed a pattern that was extremely interesting, and that the peer count on one of my nodes dropped sharply, and as the peer count dropped, so did the number of goroutines running on that node:

![](https://ipfs.io/ipfs/QmeaG1dTBww9bBobQtyngH5mMG4SVswqefG7nCZpxnwTap)

I'm unsure what the underlying issue would be, but it appears that the more peers you are connected too, the more goroutines you are running, which on the surface looks to be what was causing the poor performance for my nodes.

System Specs:
```
CPU: E5-2680 v2 - 12 cores
Memory: 16GB DDR3
Disk: 750GB 10K RPM
Disk Format: ext4
IPFS Repo: BadgerDS
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.