dragonflydb / dragonflydb/dragonfly

Dragonfly sometimes fails with OOM even when `--cache_mode=true`

Open
#3,155 7 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

**Describe the bug**
Dragonfly fails with OOM even when `--cache_mode=true`. This happens after multiple hours/days of intense load.

**To Reproduce**
We noticed this on several services that were processing ~200k commands/s and which had between 11k and 15k clients. Also, at the time of the OOM restart, the RSS memory was 162GB, while the used memory was 128GB.

```
Jun 08 13:56:10 dragonfly-1-4 systemd[1]: dragonfly.service: A process of this unit has been killed by the OOM killer.
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Main process exited, code=killed, status=9/KILL
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Failed with result 'oom-kill'.
Jun 08 13:56:38 dragonfly-1-4 systemd[1]: dragonfly.service: Consumed 2w 4d 4h 34min 1.671s CPU time.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: dragonfly.service: Scheduled restart job, restart counter is at 1.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: Stopped dragonfly.service - Aiven dragonfly in container.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: dragonfly.service: Consumed 2w 4d 4h 34min 1.671s CPU time.
Jun 08 13:56:53 dragonfly-1-4 systemd[1]: Started dragonfly.service - Aiven dragonfly in container.
```

**Expected behavior**
Dragonfly shouldn't crash.

**Environment (please complete the following information):**
- OS: Fedora 38
- Kernel: 6.6.12
- Containerized?: Fedora toolbox container
- Dragonfly Version: 1.18.1

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by reproducing the OOM under the reported ~200k commands/s and 11k–15k clients with `--cache_mode=true`, then compare RSS with used memory; done means identifying and fixing the excess memory growth and adding a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.