influxdata / influxdata/influxdb

Compactions causing iowait

Open
#24,063 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Our Influx db starts 8 compactions at a time which uses up all the memory and causes cpu iowait. The iowait is not induced by the disk and we do see swapping on the instance when this starts. We are aware of the series cardinality issues on our database, is it caused by this ? if not then is there a way to make the compactions less intense. So far I have tried to use max-concurrent-compactions and compact-throughput-* but to no avail. Can someone please advise what we are doing wrong here ? Meanwhile I will try to upgrade to the latest minor version.

Linux 3.16.0-5-amd64 x86_64
Influx version: v1.8.6
[block.txt](https://github.com/influxdata/influxdb/files/10500006/block.txt)
[diagnostics.txt](https://github.com/influxdata/influxdb/files/10500008/diagnostics.txt)
[goroutine.txt](https://github.com/influxdata/influxdb/files/10500009/goroutine.txt)
[iostat.csv](https://github.com/influxdata/influxdb/files/10500117/iostat.csv)
[profiles.tar.gz](https://github.com/influxdata/influxdb/files/10500118/profiles.tar.gz)
[shards.txt](https://github.com/influxdata/influxdb/files/10500120/shards.txt)
[stats.txt](https://github.com/influxdata/influxdb/files/10500121/stats.txt)
[vars.txt](https://github.com/influxdata/influxdb/files/10500122/vars.txt)
[heap.txt.zip](https://github.com/influxdata/influxdb/files/10500123/heap.txt.zip)

[logs.csv](https://github.com/influxdata/influxdb/files/10500012/logs.csv)
[config.txt](https://github.com/influxdata/influxdb/files/10500031/config.txt)
Screen Shot 2023-01-25 at 3 10 17 PM
Screen Shot 2023-01-25 at 3 07 38 PM

Contributor guide

Open the contributing guide

Research direction

Review config.txt alongside diagnostics.txt, stats.txt, and the attached profiles and logs to understand the compaction and memory behavior. Compare the reported settings with the InfluxDB v1.8.6 compaction configuration and determine whether the evidence supports a product bug or a tuning issue. Done means documenting the cause and a reproducible, supported mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.