influxdata / influxdata/influxdb

Full compact memory limit

Open
#24,191 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
In my environment every shard full compact cause OS memory increase too much to trigger oom-killer

__Current behavior:__
![image](https://user-images.githubusercontent.com/9251710/230840290-7670c580-8cd3-4e59-9d20-2fc3c657d451.png)

`compact-full-write-cold-duration=4h`, `shard_duration=7d`. My os timezone is `+08:00`, so you can see memory increase
at 12:00 every monday. (My influxd process use memory 80% normally)

__Desired behavior:__
Just like `compact-throughput` could limit compact write io. Memory used should be limited when compacting full level.

Contributor guide

Open the contributing guide

Research direction

No file or test entry point is named in the issue. Start by tracing the full-level compaction path and comparing it with compact-throughput; done means full compaction has a memory limit that prevents the reported OS-memory increase.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.