influxdata / influxdata/influxdb

High CPU consumption with InfluxDB 1.7.9

Open
#15,627 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hi,

We're using InfluxDB with telegraf on a monitoring server. Only telegraf database is present in InfluxDB.
With telegraf, we use the 'tail' plugin to insert data from a file using InfluxDB format. All data is stored in measurement "loopingd"
We have a script to check data present in influxDB. This script run the following query very regularly:

```
select count(rtt), mean(rtt) from loopingd where server='$server' and time >= '$start_ts' and time < '$end_ts' group by time(1d)
```

Where:
- `$server` is the server we want to check
- `$start_ts` is a timestamp in nanoseconds corresponding to previous minute, rounded to the minute (exemple: if now is 15:30:34, this will be the timestamp for 15:29:00).
- `$end_ts`is a timestamp in nanoseconds corresponding to current minute, rounded to the minute (exemple: if now is 15:30:34, this will be the timestamp for 15:30:00).

We currently have 24 097 076 points stored in this measurement.

After upgrading InfluxDB to v1.7.9 we observed an increase in CPU usage from 15% usage to 53% usage.
When doing a `top` command, we saw a lot of cpu consummed by the queries above.

__Steps to reproduce:__
I'm not sure on how to reproduce the issue, so I will try to explain how to create a setup like mine.
1. Install InfluxDB 1.7.8
2. Create a database named telegraf
3. Insert 50 entries in a measurement `loopingd` with following labes: `host`, `path`, `server` and a field named `rtt`
4. Perform multiple queries `select count(rtt), mean(rtt) from loopingd where server='$server' and time >= '$start_ts' and time < '$end_ts' group by time(1d)` and measure cpu usage on server
5. Upgrade to InfluxDB 1.7.9
6. Execute the same queries and measure CPU usage

__Expected behavior:__
CPU usage is the same between point 4 and 6

__Actual behavior:__
CPU usage is higher at point 6 than at point 4.

__Environment info:__

* System info: Run `uname -srm` and copy the output here
```
Linux 4.4.0-1096-aws x86_64
```
* InfluxDB version: Run `influxd version` and copy the output here
```
InfluxDB v1.7.9 (git: 1.7 23bc63d43a8dc05f53afa46e3526ebb5578f3d88)
```
* Other relevant environment details: Container runtime, disk info, etc
```
N/A
```

__Config:__
Copy any non-default config values here or attach the full config as a gist or file.

We use the default configuration file.

__Logs:__
Include snippet of errors in log.

None provided

__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt

# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```
[profiles.tar.gz](https://github.com/influxdata/influxdb/files/3784165/profiles.tar.gz)
[vars.txt](https://github.com/influxdata/influxdb/files/3784168/vars.txt)
[iostat.txt](https://github.com/influxdata/influxdb/files/3784178/iostat.txt)

Contributor guide

Open the contributing guide

Research direction

Start by comparing the repeated loopingd queries on InfluxDB 1.7.8 and 1.7.9 using the reproduction steps and the attached profiles.tar.gz, vars.txt, and iostat.txt. Investigate the CPU difference for the time-bounded aggregate query filtered by server; done means identifying and correcting the 1.7.9 regression so CPU usage matches the earlier version.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.