dragonflydb / dragonflydb/dragonfly
Dragonfly sometimes fails with OOM even when `--cache_mode=true`
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
**Describe the bug**
Dragonfly fails with OOM even when `--cache_mode=true`. This happens after multiple hours/days of intense load.
**To Reproduce**
We noticed this on several services that were processing ~200k commands/s and which had between 11k and 15k clients. Also, at the time of the OOM restart, the RSS memory was 162GB, while the used memory was 128GB.
```
Jun 08 13:56:10 dragonfly-1-4 systemd[1]: dragonfly.service: A process of this unit has been killed by the OOM killer.
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Main process exited, code=killed, status=9/KILL
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Failed with result 'oom-kill'.
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Consumed 2w 4d 4h 34min 1.671s CPU time.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: dragonfly.service: Scheduled restart job, restart counter is at 1.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: Stopped dragonfly.service - Aiven dragonfly in container.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: dragonfly.service: Consumed 2w 4d 4h 34min 1.671s CPU time.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: Started dragonfly.service - Aiven dragonfly in container.
```
**Expected behavior**
Dragonfly shouldn't crash.
**Environment (please complete the following information):**
- OS: Fedora 38
- Kernel: 6.6.12
- Containerized?: Fedora toolbox container
- Dragonfly Version: 1.18.1
Contributor guide
Research direction
No source file, test, or entry point is named. Start by reproducing the OOM under the reported ~200k commands/s and 11k–15k clients with `--cache_mode=true`, then compare RSS with used memory; done means identifying and fixing the excess memory growth and adding a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100