airbytehq / airbytehq/airbyte

OOM in replication jobs on newer redhat-based Linux kernels

オープン
#61,698 コメント 5 件 リアクション 7 件 担当者 0 名 GitHub で見る
area/platform community team/use
主要言語
Python
スター
22.1k
フォーク
5.3k
PR マージ指標
PR 指標を取得中

説明

### Topic

_No response_

### Relevant information

After upgrading some of our kubernetes nodes to kernel 6.12, we discovered that any replication pod running on these nodes were being killed due to OOM errors. This seems to be related to https://bugs.openjdk.org/browse/JDK-8347129

In summary: The newer kernels no longer include the cpuset cgroup which OpenJDK uses to determine the available heap space. The JDK falls back to host resources, seeing all the memory on the host rather than what's available to the pod. Since the replication jobs specify -XX:MaxRAMPercentage=75.0, it sets max heap far above the pod's limit.

While the replication pod was running, I was able to exec in and check the memory settings:
```
[airbyte@replication-job-36101-attempt-0 app]$ java -XX:MaxRAMPercentage=75 -XshowSettings -Xlog:os+container=debug --version
[0.001s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] controller cpu is not enabled

[0.001s][debug][os,container] One or more required controllers disabled at kernel level.
VM settings:
Max. Heap Size (Estimated): 93.99G
Using VM: OpenJDK 64-Bit Server VM
```
The replication pod had a 2G limit set in kubernetes so it quickly OOM'd.

I've tested with airbyte versions 1.5.4, 1.6.2 and 1.7.0 against kernels 6.12, 6.12, 6.13 and 6.14 and all experienced this problem.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。