influxdata / influxdata/influxdb
InfluxDB: High CPU utilization in v.1.8.1 & v.1.8.2
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Experienced performance issues with InfluxDB after upgrading from InfluxDB v.1.8.0 to v1.8.1 or v.1.8.2 Currently the problem is temporarily handled by downgrading back to 1.8.0.
__Steps to reproduce:__
In this environment, nothing else than
1. Upgrade InfluxDB version to 1.8.1 or 1.8.2
__Expected behavior:__
InfluxDB updates to a newer version and continues to work without issues.
__Actual behavior:__
CPU Utilization spiking to ~100%, some databases are being queried successfully, where as at least one of the largest databases (~65GB) is returning HTTP POST 500 / timeout.
Other metrics, such as memory & disk metrics are not affected drastically.
Screenshots from Grafana to visualize the CPU utilization after upgrading to v.1.8.1

__Environment info:__
* System info: Linux 3.10.0-1127.18.2.el7.x86_64 x86_64
* InfluxDB version: InfluxDB v1.8.0 (git: 1.8 781490d)
* Host VM specs:
CentOS 7
8 vCPUs
52 GB RAM
500 GB SSD disk
* Other environment details:
No other heavy workloads running on the servers other than InfluxDB. Grafana front-end.
~65 databases
~150 GB of data (~85% raw data, ~15% downsampled)
Config settings on default other than some directory settings and TSI indexing turned on
Wal and Data directories are located on the same storage device
__Logs:__
Example error line from journalctl:
```
Sep 10 10:01:04 influxdb02 influxd[22667]: ts=2020-09-10T07:01:04.768040Z lvl=error msg="[500] - \"timeout\"" log_id=0P9h7VtW000 service=httpd
```
Example error line from HTTP access log:
```
x.x.x.x - telegraf [10/Sep/2020:10:01:00 +0300] "POST /write?db=all_operated HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 62720985-f333-11ea-ac27-42010ae8030c 10609871
```
__Other notes:__
CPU Load is also constantly high, these issues are most likely somehow linked together.

There was memory issues caused by CQs too, but disabling CQ:s on the largest database resolved them. CPU utilization & load was not affected by this.
Retention policies in use:
```
> show retention policies
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 false
raw 336h0m0s 24h0m0s 1 true
agg 9600h0m0s 168h0m0s 1 false
```
Data is downsampled from "raw" to "agg" RP with continuous queries.
```
> show continuous queries
name:
name query
---- -----
cq_aggregate CREATE CONTINUOUS QUERY cq_aggregate ON BEGIN SELECT mean(*) INTO .agg.:MEASUREMENT FROM .raw./.*/ GROUP BY time(5m), * END
```
Contributor guide
Research direction
Start by reproducing the upgrade from InfluxDB 1.8.0 to 1.8.1 or 1.8.2 in the reported Linux environment, then inspect the journalctl and HTTP access logs while comparing CPU, load, and timeout behavior. Done means identifying the regression and confirming that upgrades no longer cause CPU spikes or HTTP 500/timeouts for the affected database.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, linux
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100