influxdata / influxdata/influxdb

InfluxDB: High CPU utilization in v.1.8.1 & v.1.8.2

Open
#19,543 28 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Experienced performance issues with InfluxDB after upgrading from InfluxDB v.1.8.0 to v1.8.1 or v.1.8.2 Currently the problem is temporarily handled by downgrading back to 1.8.0.

__Steps to reproduce:__
In this environment, nothing else than
1. Upgrade InfluxDB version to 1.8.1 or 1.8.2

__Expected behavior:__
InfluxDB updates to a newer version and continues to work without issues.

__Actual behavior:__
CPU Utilization spiking to ~100%, some databases are being queried successfully, where as at least one of the largest databases (~65GB) is returning HTTP POST 500 / timeout.
Other metrics, such as memory & disk metrics are not affected drastically.

Screenshots from Grafana to visualize the CPU utilization after upgrading to v.1.8.1
![v1 8 1_cpu_util](https://user-images.githubusercontent.com/70326653/92712755-c8d57b80-f362-11ea-89e8-76220dc73e01.JPG)

__Environment info:__

* System info: Linux 3.10.0-1127.18.2.el7.x86_64 x86_64
* InfluxDB version: InfluxDB v1.8.0 (git: 1.8 781490d)
* Host VM specs:
CentOS 7
8 vCPUs
52 GB RAM
500 GB SSD disk
* Other environment details:
No other heavy workloads running on the servers other than InfluxDB. Grafana front-end.
~65 databases
~150 GB of data (~85% raw data, ~15% downsampled)
Config settings on default other than some directory settings and TSI indexing turned on
Wal and Data directories are located on the same storage device

__Logs:__
Example error line from journalctl:
```
Sep 10 10:01:04 influxdb02 influxd[22667]: ts=2020-09-10T07:01:04.768040Z lvl=error msg="[500] - \"timeout\"" log_id=0P9h7VtW000 service=httpd
```

Example error line from HTTP access log:
```
x.x.x.x - telegraf [10/Sep/2020:10:01:00 +0300] "POST /write?db=all_operated HTTP/1.1" 500 20 "-" "Go-http-client/1.1" 62720985-f333-11ea-ac27-42010ae8030c 10609871
```

__Other notes:__
CPU Load is also constantly high, these issues are most likely somehow linked together.
![v 1 8 0_load](https://user-images.githubusercontent.com/70326653/93427853-2fa8e680-f8c7-11ea-97eb-680da69301db.JPG)

There was memory issues caused by CQs too, but disabling CQ:s on the largest database resolved them. CPU utilization & load was not affected by this.
Retention policies in use:
```
> show retention policies
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 false
raw 336h0m0s 24h0m0s 1 true
agg 9600h0m0s 168h0m0s 1 false
```
Data is downsampled from "raw" to "agg" RP with continuous queries.
```
> show continuous queries
name:
name query
---- -----
cq_aggregate CREATE CONTINUOUS QUERY cq_aggregate ON BEGIN SELECT mean(*) INTO .agg.:MEASUREMENT FROM .raw./.*/ GROUP BY time(5m), * END
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the upgrade from InfluxDB 1.8.0 to 1.8.1 or 1.8.2 in the reported Linux environment, then inspect the journalctl and HTTP access logs while comparing CPU, load, and timeout behavior. Done means identifying the regression and confirming that upgrades no longer cause CPU spikes or HTTP 500/timeouts for the affected database.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, linux
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.