[Memory Leak] Worker pod keep consuming memory over time
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Helm Chart Version
1.2.0
### What step the error happened?
Other
### Relevant information
Hi there!
I've been running Airbyte 1.2.0 on EKS over the last 3-4 months, with karpenter as the node provisioner. Every platform component runs in on-demand instances (in the same NodePool), and the jobs run in spot instances.
These are the worker values:
```yaml
## @section Worker Parameters
worker:
# -- Add extra annotations to the worker pods
podAnnotations:
reloader.stakater.com/auto: "true"
livenessProbe:
# -- Initial delay seconds for livenessProbe
initialDelaySeconds: 60
# -- Period seconds for livenessProbe
periodSeconds: 20
# -- Failure threshold for livenessProbe
failureThreshold: 5
readinessProbe:
# -- Initial delay seconds for readinessProbe
initialDelaySeconds: 60
# -- Period seconds for readinessProbe
periodSeconds: 20
# -- Failure threshold for readinessProbe
failureThreshold: 5
## worker resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
resources:
requests:
cpu: 300m
memory: 1Gi
limits:
cpu: 300m
memory: 1Gi
log:
level: "DEBUG"
extraEnv:
- name: MAX_CHECK_WORKERS
value: "30"
- name: MAX_SYNC_WORKERS
value: "30"
- name: DISCOVER_REFRESH_WINDOW_MINUTES
value: "240" # 4 hours
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_MAX_TIMEOUT_DAYS
value: "1"
hpa:
enabled: true
debug:
enabled: true
```
Due to some issues (like [this one](https://github.com/airbytehq/airbyte/issues/48879)), I deployed a PDB that prevents karpenter from rotating the worker pod when consolidating nodes. Since then, I noticed a memory leak in the worker pod, as can be seen here (worker pod memory consumption over time, and its requests and limits):

If I restart the deployment, memory consumption drops to 500-600 MB, increasing over time again until it's OOM killed.
I've been searching around old issues related to memory leaks, and I understood it was a solved issue. Also, I tried running the garbage collector manually in the worker pod by running `jcmd PID GC.run`, but didn't solve the problem.
Can you please help me understand why this is happening?
Thanks!
### Relevant log output
```shell
```
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.