argoproj / argoproj/argo-workflows
Argoexec init contariner meet OOMKilled issue
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))
### What happened/what you expected to happen?
The argoexec init container OOMKilled.
I think that 64Mi memory is enough for init container.
We are looking for the right limit to set and that we check the differences between environments but we cannot correlate.
Need argo workflow team to check why argoexec init container use more memory than 64Mi. it's abnormal.
And which cause the init container use more memory?
```
Init Containers:
init:
Container ID: containerd://3eb8993d0faaec1cd0862bde4e92e33558ca6e329e2e0c960d101a69ad4cc22b
Image: quay.io/argoproj/argoexec:latest
Image ID: quay.io/argoproj/argoexec@sha256:5cd920d5e57cdc0881f13a9b39238cea462eee59ea35a39f1142b7994a27a5fd
Port:
Host Port:
Command:
argoexec
init
--loglevel
debug
--log-format
text
State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 12 Jun 2023 16:19:37 +0800
Finished: Mon, 12 Jun 2023 16:19:38 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 64Mi
Requests:
cpu: 100m
memory: 64Mi
```
### Version
V3.4.8 or lastest
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
argo workflow configmap set as this:
kubectl get cm workflow-controller-configmap -o yaml -n argo
apiVersion: v1
data:
executor: |
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 0.1
memory: 64Mi
limits:
cpu: 0.5
memory: 64Mi
args:
- --loglevel
- debug
- --gloglevel
- "6"
```
### Logs from the workflow controller
```text
time="2023-06-16T06:01:51.629Z" level=info msg="node changed" namespace=XXXXX
new.message="OOMKilled (exit code 137)" new.phase=Failed new.progress=0/1 nodeID=XXXXXX old.message= old.phase=Pending old.progress=0/1 workflow=XXXXXX
time="2023-06-16T06:01:51.631Z" level=info msg="node XXXXXX message: OOMKilled (exit code 137)" namespace=XXX workflow=XXX.." ... level=deb type: 'Warning' reason: 'WorkflowNodeFailed' Failed node XXXXXXX: OOMKilled (exit code 137)"
```
### Logs from in your workflow's wait container
```text
k logs podname -n xxx -c init
time="2023-06-09T08:32:45.385Z" level=info msg="Starting Workflow Executor" version=untagged
time="2023-06-09T08:32:45.454Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-06-09T08:32:45.454Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false 。。。,Platform:linux/amd64,}"
and no other log to print.(because print this pod is OOMKilled)
```
Contributor guide
Research direction
Start with the argoexec init command and the executor resources in workflow-controller-configmap, then reproduce the OOMKilled result with the supplied 64Mi limit. Review the init-container startup logs and compare behavior across environments. Done means identifying why startup exceeds 64Mi and documenting or implementing the appropriate resource handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100