OOM in replication jobs on newer redhat-based Linux kernels
- 主要言語
- Python
- スター
- 22.1k
- フォーク
- 5.3k
- PR マージ指標
- PR 指標を取得中
説明
### Topic
_No response_
### Relevant information
After upgrading some of our kubernetes nodes to kernel 6.12, we discovered that any replication pod running on these nodes were being killed due to OOM errors. This seems to be related to https://bugs.openjdk.org/browse/JDK-8347129
In summary: The newer kernels no longer include the cpuset cgroup which OpenJDK uses to determine the available heap space. The JDK falls back to host resources, seeing all the memory on the host rather than what's available to the pod. Since the replication jobs specify -XX:MaxRAMPercentage=75.0, it sets max heap far above the pod's limit.
While the replication pod was running, I was able to exec in and check the memory settings:
```
[airbyte@replication-job-36101-attempt-0 app]$ java -XX:MaxRAMPercentage=75 -XshowSettings -Xlog:os+container=debug --version
[0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] controller cpu is not enabled
[0.001s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 93.99G
Using VM: OpenJDK 64-Bit Server VM
```
The replication pod had a 2G limit set in kubernetes so it quickly OOM'd.
I've tested with airbyte versions 1.5.4, 1.6.2 and 1.7.0 against kernels 6.12, 6.12, 6.13 and 6.14 and all experienced this problem.
コントリビューションガイド
評価
この issue はまだ評価されていません。