[ML] Don't try to change the OOM score adjustment when running under Kubernetes
- Dominant language
- C++
- Stars
- 157
- Forks
- 67
- Avg merge
- 12h 48m
- Merged PRs (30d)
- 16
Description
Kubernetes stops processes from changing their OOM score adjustment. There is more detail about this in https://github.com/kubernetes/kubernetes/issues/90973.
This causes the ML processes to log warning messages when running under Kubernetes, which can worry users. To avoid this we should change our code to somehow detect that it is running under Kubernetes and then not attempt to change the OOM score adjustment. We should also keep an eye on https://github.com/kubernetes/kubernetes/issues/90973 to see if this policy of Kubernetes ever changes.
The reason the ML processes increase their OOM score adjustment is similar to the reasoning for PostgreSQL: if the OOM killer has to choose a process it's less catastrophic for it to choose an ML job process than the Elasticsearch JVM.
Assuming our memory accounting is accurate the OOM killer is less likely to be a problem in a container that's dedicated to running Elasticsearch than on a general machine, because there's no chance that the user starts up some unrelated process that consumes a lot of memory. Therefore this adjustment is less important in containers than on general purpose machines, such as development laptops. It only really matters in a container if our memory accounting is wrong under some circumstance, and this makes a job use more memory than its model memory limit plus allowed process overhead.
Contributor guide
Assessment
This issue has not been assessed yet.