influxdata / influxdata/influxdb

🔥 influxd burns extraordinary high CPU time idling doing nothing 🔥

Open
#24,305 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

`influxd` process burns extraordinary amount of CPU time idling doing nothing, while there are no write or read requests at all. It burns more CPU than all other processes combined.

`strace`ing `influxd` process reveals that there is a swarm of `influxd` processes/threads doing 1-second waits on a futex, timing out, polling sockets and repeating the wait. `influxd` should probably use an `eventfd` instead of a mutex to avoid having to use 2 waiting APIs with 1-second polling. I may be wrong.

For my use case, I have to start `influxd` on each request and stop it after to avoid `influxd` process running all the time burning CPU needlessly.

__Steps to reproduce:__
1. Create a bucket with infinite retention.
2. Fill the bucket with 5-year's worth of 1-minute OHLCV-like data for one "measurement". That is, there are 5 floating point columns O,H,L,C,V updated every minute per "measurement". Generate this shape and type of data and insert that into the bucket using your favourite programming language.
3. Observe `influxd` process burning CPU cycles idling doing nothing, without any write or read requests at all.

__Expected behavior:__
`influxd` process burns 0 CPU cycles idling doing nothing.

__Actual behavior:__
`influxd` process burns more CPU cycles idling doing nothing than all other processes combined:
```
max@supernova:~$ ps -efww --sort -cputime
UID PID PPID C STIME TTY TIME CMD
influxdb 3690 1 3 Jun23 ? 00:43:47 /usr/bin/influxd
max 4871 4642 0 Jun23 ? 00:06:37 /usr/bin/plasmashell --no-respawn
max 5103 5027 0 Jun23 ? 00:04:24 /usr/bin/nvidia-smi dmon -d 2 -s pucm
root 1714 2 0 Jun23 ? 00:03:59 [irq/168-nvidia]
max 4833 4642 0 Jun23 ? 00:03:43 /usr/bin/kwin_x11 --replace
max 15062 15035 1 Jun23 ? 00:02:16 /opt/google/chrome/chrome --type=gpu-process ...
max 5027 4642 0 Jun23 ? 00:01:47 /usr/bin/ksystemstats
root 2888 2740 0 Jun23 tty1 00:01:45 /usr/lib/xorg/Xorg -nolisten tcp ...
...
```
In the above listing, `influxd` burnt 43 minutes of CPU doing nothing during the 24 hours since the boot.

__Environment info:__

* System info: `Linux 5.19.0-1025-lowlatency x86_64`
* InfluxDB version: `InfluxDB v2.7.1 (git: 407fa622e9) build_date: 2023-04-28T13:24:27Z`
* Other relevant environment details: influxd data files reside on a SATA 6G SSD disk.

__Config:__
```
bolt-path = "/home/influxdb/.influxdb/influxd.bolt"
engine-path = "/home/influxdb/.influxdb/engine"
flux-log-enabled = true
query-concurrency = 32
query-queue-size = 32

storage-tsm-use-madv-willneed = true
storage-cache-snapshot-write-cold-duration = "24h0m0s"
storage-compact-full-write-cold-duration = "24h0m0s"
storage-retention-check-interval = "24h0m0s"
storage-shard-precreator-check-interval = "24h0m0s"
storage-max-concurrent-compactions = 1
storage-series-file-max-concurrent-snapshot-compactions = 1
```

Config options below `query-queue-size` were my fruitless attempts to make `influxd` not burn CPU cycles needlessly.

__Logs:__
[influxdb.log](https://github.com/influxdata/influxdb/files/11854037/influxdb.log)
[influxd.strace.txt](https://github.com/influxdata/influxdb/files/11854158/influxd.strace.txt) with 1-second waits on a futex.

__Performance:__
[profiles.tar.gz](https://github.com/influxdata/influxdb/files/11854024/profiles.tar.gz)
[iostat.txt](https://github.com/influxdata/influxdb/files/11854025/iostat.txt)

Contributor guide

Open the contributing guide

Research direction

The report centers on the influxd process and includes influxd.strace.txt, influxdb.log, profiles.tar.gz, and iostat.txt. Start by reviewing those artifacts and the reported futex/socket polling pattern, then reproduce the idle workload described. Done means idle influxd no longer consumes substantial CPU without read or write requests.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.