dragonflydb / dragonflydb/dragonfly

Memory accounting code: review

Open
#5,378 2 comments 0 reactions 0 assignees View on GitHub
fun
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 9h
Merged PRs (30d)
121

Description

I've been re-reading our memory accounting code to add better heuristics:

- [x] `max_memory_limit` is just a global `size_t` that can be mutably changed by the config registry 🎖
- [x] `used_mem_peak` and `rss_mem_peak` are unnecessarily global
- [x] Deny OOM:
* `GetMemoryUsage` updates every 1us, whereas global atomics are updated every ~`1/shard_count`**ms** (rss even less frequently), i.e. 99% of the calls are ineffective
* Not sure why we use command timestamps?
* No uniformity in flag access. Some code calls `GetFlag` on hot paths, server state caches at least a handful of other flags. Once calculated parameters were left forgotten with flag updates (like `soft_budget_limit_`)
* "Modules" often don't care about centralizing memory calculations. Deny oom has 3 places where it repeats its calculation (using different sources at the same time), tiering calculates it's `offload threshold` two times - same formula - differently looking code
* No real comments to clarify the meaning or usage of important variables or calculations

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.