argoproj / argoproj/argo-workflows

StartError (exit code 128) when pod tries to mount configmap that is being changed at same time

Open
#11,983 5 comments 0 reactions 0 assignees View on GitHub
area/executor P3
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

a step of my workflow mounts a configmap ie
```yaml
volumes:
- name: my-py
configMap:
name: my-py
defaultMode: 0744

volumeMounts:
- name: my-py
mountPath: /my.py
subPath: my.py
```

it just so happened that i ran a workflow at the same time as the configmap was being replaced/overwritten

the workflow failed with below message and won't show the logs of that step in the argo ui. but upon looking at the container logs from external logging tool it appears the pod ran all the code successfully.

```
StartError (exit code 128): failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/f22aa0b7-afdc-497c-beb3-ccb394e8428d/volume-subpaths/my-py/wait/9" to rootfs at "/mainctrfs/my.py": mount /var/lib/kubelet/pods/f22aa0b7-afdc-497c-beb3-ccb394e8428d/volume-subpaths/my-py/wait/9:/mainctrfs/my.py (via /proc/self/fd/6), flags: 0x5001, data: context="system_u:object_r:data_t:s0:c923,c1020": no such file or directory: unknown
```

i have run 10000s of this type of workflow successfully, it just happened that this one run above was at same time as the configmap was being updated

https://github.com/kubernetes/kubernetes/blob/2c4a863bf9b3f0d290a3f21bf7fe57d9f15a39b5/test/e2e/node/pods.go#L766-L768C66 maybe some race condition upstream

### Version

3.4.11

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
n/a
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start by reading the Kubernetes test/e2e/node/pods.go reference around lines 766-768 and compare it with the reported ConfigMap replacement timing. Use the workflow manifest and StartError output to determine whether the mount race is reproducible; done means identifying a reproducible failure and a verified way to prevent the failed step or loss of its Argo UI logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.