influxdata / influxdata/influxdb
Compactions causing iowait
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Our Influx db starts 8 compactions at a time which uses up all the memory and causes cpu iowait. The iowait is not induced by the disk and we do see swapping on the instance when this starts. We are aware of the series cardinality issues on our database, is it caused by this ? if not then is there a way to make the compactions less intense. So far I have tried to use max-concurrent-compactions and compact-throughput-* but to no avail. Can someone please advise what we are doing wrong here ? Meanwhile I will try to upgrade to the latest minor version.
Linux 3.16.0-5-amd64 x86_64
Influx version: v1.8.6
[block.txt](https://github.com/influxdata/influxdb/files/10500006/block.txt)
[diagnostics.txt](https://github.com/influxdata/influxdb/files/10500008/diagnostics.txt)
[goroutine.txt](https://github.com/influxdata/influxdb/files/10500009/goroutine.txt)
[iostat.csv](https://github.com/influxdata/influxdb/files/10500117/iostat.csv)
[profiles.tar.gz](https://github.com/influxdata/influxdb/files/10500118/profiles.tar.gz)
[shards.txt](https://github.com/influxdata/influxdb/files/10500120/shards.txt)
[stats.txt](https://github.com/influxdata/influxdb/files/10500121/stats.txt)
[vars.txt](https://github.com/influxdata/influxdb/files/10500122/vars.txt)
[heap.txt.zip](https://github.com/influxdata/influxdb/files/10500123/heap.txt.zip)
[logs.csv](https://github.com/influxdata/influxdb/files/10500012/logs.csv)
[config.txt](https://github.com/influxdata/influxdb/files/10500031/config.txt)

Contributor guide
Research direction
Review config.txt alongside diagnostics.txt, stats.txt, and the attached profiles and logs to understand the compaction and memory behavior. Compare the reported settings with the InfluxDB v1.8.6 compaction configuration and determine whether the evidence supports a product bug or a tuning issue. Done means documenting the cause and a reproducible, supported mitigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100