influxdata / influxdata/influxdb

Influxdb shr memory usage is very high, leading to oom

Open
#22,840 0 comments 0 reactions 0 assignees View on GitHub
oom
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We are now using Influxdb to store flink metrics, and the feature of metrics is that each flink task will generate a large number of series, and the flink task will also generate a new series after restarting. Due to this large number of series, our current influxdb memory usage Always high。

After our observation and analysis, after the service has been idle for a long time, the data has basically been eliminated due to the expiration mechanism, and the memory usage is still high

Use top view:RES 11G SHR 9G.

By looking at the code, we found that mmap, which is basically SeriesSegment, occupies shard memory, and some files are fully loaded in RES.

And the seriesSegement file has not been deleted, occupying the disk. According to my understanding, can't seriesSegement be deleted after compact? All previous series have been stored in the memory or disk, then the wal log from the current time should be cleared, I don't know if I understand it correctly.

__Expected behavior:__
The SeriesSegment will be deleted after compacting, and the new series log will be written into the new SeriesSegment. All the data in the old SeriesSegment will be compacted to the index, which is no longer needed.

__Actual behavior:__
Each compact action will traverse all SeriesSegments, occupying a lot of shard memory

__Environment info:__
influxdb:1.7.6
linux

__Config:__
index-version = tsi1

Contributor guide

Open the contributing guide

Research direction

Start with the TSI1 shard compaction path and its SeriesSegment mmap handling. Trace how compaction processes old SeriesSegments and whether their files are removed afterward. Done means compaction no longer unnecessarily retains or traverses obsolete segments, while preserving the expected index and new-series behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.