airbytehq / airbytehq/airbyte

[Memory Leak] Worker pod keep consuming memory over time

Offen
#56,944 6 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/platform community team/platform-move type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Helm Chart Version

1.2.0

### What step the error happened?

Other

### Relevant information

Hi there!

I've been running Airbyte 1.2.0 on EKS over the last 3-4 months, with karpenter as the node provisioner. Every platform component runs in on-demand instances (in the same NodePool), and the jobs run in spot instances.

These are the worker values:

```yaml
## @section Worker Parameters

worker:
# -- Add extra annotations to the worker pods
podAnnotations:
reloader.stakater.com/auto: "true"

livenessProbe:
# -- Initial delay seconds for livenessProbe
initialDelaySeconds: 60
# -- Period seconds for livenessProbe
periodSeconds: 20
# -- Failure threshold for livenessProbe
failureThreshold: 5

readinessProbe:
# -- Initial delay seconds for readinessProbe
initialDelaySeconds: 60
# -- Period seconds for readinessProbe
periodSeconds: 20
# -- Failure threshold for readinessProbe
failureThreshold: 5

## worker resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
resources:
requests:
cpu: 300m
memory: 1Gi
limits:
cpu: 300m
memory: 1Gi

log:
level: "DEBUG"

extraEnv:
- name: MAX_CHECK_WORKERS
value: "30"
- name: MAX_SYNC_WORKERS
value: "30"
- name: DISCOVER_REFRESH_WINDOW_MINUTES
value: "240" # 4 hours
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_SUCCESSIVE
value: "3"
- name: SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_TOTAL
value: "5"
- name: SYNC_JOB_MAX_TIMEOUT_DAYS
value: "1"

hpa:
enabled: true

debug:
enabled: true
```

Due to some issues (like [this one](https://github.com/airbytehq/airbyte/issues/48879)), I deployed a PDB that prevents karpenter from rotating the worker pod when consolidating nodes. Since then, I noticed a memory leak in the worker pod, as can be seen here (worker pod memory consumption over time, and its requests and limits):

![Image](https://github.com/user-attachments/assets/c75eef90-4989-4e31-a81a-f99611b80ded)

If I restart the deployment, memory consumption drops to 500-600 MB, increasing over time again until it's OOM killed.

I've been searching around old issues related to memory leaks, and I understood it was a solved issue. Also, I tried running the garbage collector manually in the worker pod by running `jcmd PID GC.run`, but didn't solve the problem.

Can you please help me understand why this is happening?

Thanks!

### Relevant log output

```shell

```

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.