ipfs / ipfs/kubo

requesting node metrics from the API in a short interval causes a high CPU load

Open
#7,528 3 comments 0 reactions 0 assignees View on GitHub
effort/days exp/expert kind/enhancement P3 status/ready topic/perf
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

#### Version information:
```
go-ipfs version: 0.7.0-dev
Repo version: 10
System version: amd64/linux
Golang version: go1.14.4

master@7ce1d751f
```

#### Description:
I'm running ipfs on a new server with an SSD storage. I'm writing a lot of individual files with `ipfs add --chunker 'buzhash' --cid-version 1 --hash 'blake2b-256'` to the node, copy them to the right location in the MFS and unpin them again (since ipfs files write doesn't support setting a non-standard chunker).

Afterwards, the MFS-folder-CID is pinned on ipfs-cluster, which runs on the same node.

ipfs-cluster shows that all cluster-pins are locally pinned, which are part of the pinset.

Another remote server has also all pins of the cluster set pinned, two other servers still catch up - so they are receiving blocks from the local node.

The low bandwidth use, while it should send a somewhat large folder to two other nodes brought a possible issue to my attention - the outgoing network speed was shown as around 4 MBit/s which is extremely slow for a server basically doing nothing else.

The CPU usage (around 200%) is extremely high for the network usage, so I thought it might still publish CIDs, and went to sleep.

System specs:
4 dedicated cores for the VM from an AMD EPYC 7702P 64-Core Processor; 16 GB of memory.

There are no background tasks running, just ipfs and ipfs-cluster. ipfs-cluster uses like no CPU resources at all.

I tried changing the dht type to `dhtclient`, but this resulted in no change. Restarting the service also resulted in no change, the CPU usage just jumps up again to around 200%.

The debug data (I forgot to collect the last ones) - and the binary since it's built from the master. When I read the cpu-profile right, it leads to lot of CPU-time being used by go-ds-badger and go-ipfs-blockstore and functions called by them (flame graph). The debug data was collected some minutes after a restart of the IPFS-daemon, while the ipfs-cluster-service was turned off.

[debug.tar.gz](https://github.com/ipfs/go-ipfs/files/4891882/debug.tar.gz)

Here are some performance numbers collected on the system, which basically shows no difference in load, while there's only very low network traffic.

![Screenshot_20200708_174457](https://user-images.githubusercontent.com/614929/86942172-23751000-c145-11ea-89fc-a42668495242.png)
![Screenshot_20200708_174428](https://user-images.githubusercontent.com/614929/86942175-240da680-c145-11ea-987b-2b368ab41f08.png)
![Screenshot_20200708_174318](https://user-images.githubusercontent.com/614929/86942178-24a63d00-c145-11ea-8f5a-960c39e4d532.png)
![Screenshot_20200708_171025](https://user-images.githubusercontent.com/614929/86942182-253ed380-c145-11ea-9538-0e3b7fd2ef7a.png)
![Screenshot_20200708_170937](https://user-images.githubusercontent.com/614929/86942185-253ed380-c145-11ea-8750-6e9b26738c33.png)
![Screenshot_20200708_170907](https://user-images.githubusercontent.com/614929/86942187-25d76a00-c145-11ea-8268-2690b2b80640.png)
![Screenshot_20200708_170850](https://user-images.githubusercontent.com/614929/86942189-26700080-c145-11ea-923d-5f7dabba8306.png)
![Screenshot_20200708_170803](https://user-images.githubusercontent.com/614929/86942191-27089700-c145-11ea-977c-feef240957c4.png)
![Screenshot_20200708_170230](https://user-images.githubusercontent.com/614929/86942194-27089700-c145-11ea-8a0f-baa71920e331.png)
![Screenshot_20200708_170047](https://user-images.githubusercontent.com/614929/86942195-27a12d80-c145-11ea-8ffd-9084dfcacfd7.png)
![Screenshot_20200708_165957](https://user-images.githubusercontent.com/614929/86942196-2839c400-c145-11ea-954a-5288a4b51b8f.png)
![Screenshot_20200708_165927](https://user-images.githubusercontent.com/614929/86942197-2839c400-c145-11ea-99cd-711738d528da.png)
![Screenshot_20200708_165639](https://user-images.githubusercontent.com/614929/86942200-28d25a80-c145-11ea-904d-811d10cf80b0.png)
![Screenshot_20200708_165452](https://user-images.githubusercontent.com/614929/86942201-296af100-c145-11ea-9223-900e193a07b1.png)
![Screenshot_20200708_165352](https://user-images.githubusercontent.com/614929/86942204-2a038780-c145-11ea-88cd-5bc13a1f37f7.png)
![Screenshot_20200708_165323](https://user-images.githubusercontent.com/614929/86942205-2a038780-c145-11ea-982e-0b24e24a22fa.png)

## Config

`DisableBandwidthMetrics` and `DisableNatPortMap` are true, `EnableAutoRelay` and `EnableRelayHop` are false. I use the server-profile and routing.type is `dhtclient`. I use the badgerds, `StorageGCWatermark` is 90, `StorageMax` is 280GB.

```bash
$ ipfs repo stat
NumObjects: 610148
RepoSize: 98410186788
StorageMax: 280000000000
RepoPath: /var/lib/ipfs
Version: fs-repo@10
```

I use the systemd-hardening.service file from the repo, but changed the ExecStart to

`/usr/bin/ipfs daemon --enable-gc --enable-pubsub-experiment --enable-namesys-pubsub`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.