elastic / elastic/ml-cpp

[ML] Don't try to change the OOM score adjustment when running under Kubernetes

Open
#1,899 1 comment 1 reaction 0 assignees View on GitHub
:ml
Dominant language
C++
Stars
157
Forks
67
Avg merge
12h 48m
Merged PRs (30d)
16

Description

Kubernetes stops processes from changing their OOM score adjustment. There is more detail about this in https://github.com/kubernetes/kubernetes/issues/90973.

This causes the ML processes to log warning messages when running under Kubernetes, which can worry users. To avoid this we should change our code to somehow detect that it is running under Kubernetes and then not attempt to change the OOM score adjustment. We should also keep an eye on https://github.com/kubernetes/kubernetes/issues/90973 to see if this policy of Kubernetes ever changes.

The reason the ML processes increase their OOM score adjustment is similar to the reasoning for PostgreSQL: if the OOM killer has to choose a process it's less catastrophic for it to choose an ML job process than the Elasticsearch JVM.

Assuming our memory accounting is accurate the OOM killer is less likely to be a problem in a container that's dedicated to running Elasticsearch than on a general machine, because there's no chance that the user starts up some unrelated process that consumes a lot of memory. Therefore this adjustment is less important in containers than on general purpose machines, such as development laptops. It only really matters in a container if our memory accounting is wrong under some circumstance, and this makes a job use more memory than its model memory limit plus allowed process overhead.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.