cockroachdb / cockroachdb/cockroach

operation/failureinjection: OOM

Open
#149,161 7 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations C-enhancement E-starter T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

There are many failure injection techniques to induce an OOM. E.g., stress-ng, cgroup memory.max, and GOMEMLIMIT. However, these techniques are _out-of-process_, making them somewhat synthetic and side-effect free. E.g.,

```
sudo bash -c "echo -1000 > /proc/self/oom_score_adj && exec stress-ng --vm 2 --vm-bytes 20G --vm-keep --vm-populate --vm-locked --timeout 0 --no-oom-adjust"
```

induces memory pressure and makes itself immune to the OOM killer. It will most likely cause the crdb process to OOM; you can adjust `--vm-bytes` after consulting `free -h`.

Instead, using an _in-process_ method to induce OOM is arguably more realistic. It is likely to have other side-effects. E.g., a resumable job which induces an OOM could affect availability of several nodes.

Jira issue: CRDB-52023

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reading the issue and linked Jira issue CRDB-52023, then compare the cited out-of-process techniques with the requested in-process OOM approach; done should be a defined, realistic failure-injection method with its side effects and scope understood.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.