airbytehq / airbytehq/airbyte

OOM in replication jobs on newer redhat-based Linux kernels

未關閉
#61,698 5 則留言 7 個 reaction 已指派 0 人 在 GitHub 檢視
area/platform community team/use
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Topic

_No response_

### Relevant information

After upgrading some of our kubernetes nodes to kernel 6.12, we discovered that any replication pod running on these nodes were being killed due to OOM errors. This seems to be related to https://bugs.openjdk.org/browse/JDK-8347129

In summary: The newer kernels no longer include the cpuset cgroup which OpenJDK uses to determine the available heap space. The JDK falls back to host resources, seeing all the memory on the host rather than what's available to the pod. Since the replication jobs specify -XX:MaxRAMPercentage=75.0, it sets max heap far above the pod's limit.

While the replication pod was running, I was able to exec in and check the memory settings:
```
[airbyte@replication-job-36101-attempt-0 app]$ java -XX:MaxRAMPercentage=75 -XshowSettings -Xlog:os+container=debug --version
[0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] controller cpu is not enabled

[0.001s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 93.99G
Using VM: OpenJDK 64-Bit Server VM
```
The replication pod had a 2G limit set in kubernetes so it quickly OOM'd.

I've tested with airbyte versions 1.5.4, 1.6.2 and 1.7.0 against kernels 6.12, 6.12, 6.13 and 6.14 and all experienced this problem.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。