airbytehq / airbytehq/airbyte

[Memory Leak] Worker pod keep consuming memory over time

未關閉
#56,944 6 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/platform community team/platform-move type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Helm Chart Version

1.2.0

### What step the error happened?

Other

### Relevant information

Hi there!

I've been running Airbyte 1.2.0 on EKS over the last 3-4 months, with karpenter as the node provisioner. Every platform component runs in on-demand instances (in the same NodePool), and the jobs run in spot instances.

These are the worker values:

```yaml
## @section Worker Parameters

worker:
# -- Add extra annotations to the worker pods
podAnnotations:
reloader.stakater.com/auto: "true"

livenessProbe:
# -- Initial delay seconds for livenessProbe
initialDelaySeconds: 60
# -- Period seconds for livenessProbe
periodSeconds: 20
# -- Failure threshold for livenessProbe
failureThreshold: 5

readinessProbe:
# -- Initial delay seconds for readinessProbe
initialDelaySeconds: 60
# -- Period seconds for readinessProbe
periodSeconds: 20
# -- Failure threshold for readinessProbe
failureThreshold: 5

## worker resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
resources:
requests:
cpu: 300m
memory: 1Gi
limits:
cpu: 300m
memory: 1Gi

log:
level: "DEBUG"

extraEnv:
- name: MAX_CHECK_WORKERS
value: "30"
- name: MAX_SYNC_WORKERS
value: "30"
- name: DISCOVER_REFRESH_WINDOW_MINUTES
value: "240" # 4 hours
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_MAX_TIMEOUT_DAYS
value: "1"

hpa:
enabled: true

debug:
enabled: true
```

Due to some issues (like [this one](https://github.com/airbytehq/airbyte/issues/48879)), I deployed a PDB that prevents karpenter from rotating the worker pod when consolidating nodes. Since then, I noticed a memory leak in the worker pod, as can be seen here (worker pod memory consumption over time, and its requests and limits):

![Image](https://github.com/user-attachments/assets/c75eef90-4989-4e31-a81a-f99611b80ded)

If I restart the deployment, memory consumption drops to 500-600 MB, increasing over time again until it's OOM killed.

I've been searching around old issues related to memory leaks, and I understood it was a solved issue. Also, I tried running the garbage collector manually in the worker pod by running `jcmd PID GC.run`, but didn't solve the problem.

Can you please help me understand why this is happening?

Thanks!

### Relevant log output

```shell

```

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。