argoproj / argoproj/argo-workflows
StartError (exit code 128) when pod tries to mount configmap that is being changed at same time
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))
### What happened/what you expected to happen?
a step of my workflow mounts a configmap ie
```yaml
volumes:
- name: my-py
configMap:
name: my-py
defaultMode: 0744
volumeMounts:
- name: my-py
mountPath: /my.py
subPath: my.py
```
it just so happened that i ran a workflow at the same time as the configmap was being replaced/overwritten
the workflow failed with below message and won't show the logs of that step in the argo ui. but upon looking at the container logs from external logging tool it appears the pod ran all the code successfully.
```
StartError (exit code 128): failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/f22aa0b7-afdc-497c-beb3-ccb394e8428d/volume-subpaths/my-py/wait/9" to rootfs at "/mainctrfs/my.py": mount /var/lib/kubelet/pods/f22aa0b7-afdc-497c-beb3-ccb394e8428d/volume-subpaths/my-py/wait/9:/mainctrfs/my.py (via /proc/self/fd/6), flags: 0x5001, data: context="system_u:object_r:data_t:s0:c923,c1020": no such file or directory: unknown
```
i have run 10000s of this type of workflow successfully, it just happened that this one run above was at same time as the configmap was being updated
https://github.com/kubernetes/kubernetes/blob/2c4a863bf9b3f0d290a3f21bf7fe57d9f15a39b5/test/e2e/node/pods.go#L766-L768C66 maybe some race condition upstream
### Version
3.4.11
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
```YAML
n/a
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start by reading the Kubernetes test/e2e/node/pods.go reference around lines 766-768 and compare it with the reported ConfigMap replacement timing. Use the workflow manifest and StartError output to determine whether the mount race is reproducible; done means identifying a reproducible failure and a verified way to prevent the failed step or loss of its Argo UI logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100