influxdata / influxdata/influxdb
Influxdb shr memory usage is very high, leading to oom
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
We are now using Influxdb to store flink metrics, and the feature of metrics is that each flink task will generate a large number of series, and the flink task will also generate a new series after restarting. Due to this large number of series, our current influxdb memory usage Always high。
After our observation and analysis, after the service has been idle for a long time, the data has basically been eliminated due to the expiration mechanism, and the memory usage is still high
Use top view:RES 11G SHR 9G.
By looking at the code, we found that mmap, which is basically SeriesSegment, occupies shard memory, and some files are fully loaded in RES.
And the seriesSegement file has not been deleted, occupying the disk. According to my understanding, can't seriesSegement be deleted after compact? All previous series have been stored in the memory or disk, then the wal log from the current time should be cleared, I don't know if I understand it correctly.
__Expected behavior:__
The SeriesSegment will be deleted after compacting, and the new series log will be written into the new SeriesSegment. All the data in the old SeriesSegment will be compacted to the index, which is no longer needed.
__Actual behavior:__
Each compact action will traverse all SeriesSegments, occupying a lot of shard memory
__Environment info:__
influxdb:1.7.6
linux
__Config:__
index-version = tsi1
Contributor guide
Research direction
Start with the TSI1 shard compaction path and its SeriesSegment mmap handling. Trace how compaction processes old SeriesSegments and whether their files are removed afterward. Done means compaction no longer unnecessarily retains or traverses obsolete segments, while preserving the expected index and new-series behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100