OOM in replication jobs on newer redhat-based Linux kernels
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
### Topic
_No response_
### Relevant information
After upgrading some of our kubernetes nodes to kernel 6.12, we discovered that any replication pod running on these nodes were being killed due to OOM errors. This seems to be related to https://bugs.openjdk.org/browse/JDK-8347129
In summary: The newer kernels no longer include the cpuset cgroup which OpenJDK uses to determine the available heap space. The JDK falls back to host resources, seeing all the memory on the host rather than what's available to the pod. Since the replication jobs specify -XX:MaxRAMPercentage=75.0, it sets max heap far above the pod's limit.
While the replication pod was running, I was able to exec in and check the memory settings:
```
[airbyte@replication-job-36101-attempt-0 app]$ java -XX:MaxRAMPercentage=75 -XshowSettings -Xlog:os+container=debug --version
[0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] controller cpu is not enabled
[0.001s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 93.99G
Using VM: OpenJDK 64-Bit Server VM
```
The replication pod had a 2G limit set in kubernetes so it quickly OOM'd.
I've tested with airbyte versions 1.5.4, 1.6.2 and 1.7.0 against kernels 6.12, 6.12, 6.13 and 6.14 and all experienced this problem.
貢獻指南
評估
這個 Issue 還沒有評估資料。