argoproj / argoproj/argo-workflows
PDB not created when Workflow waiting on lock
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened/what did you expect to happen?
This is a follow-up to my findings in https://github.com/argoproj/argo-workflows/pull/6356#discussion_r1574367137 / https://github.com/argoproj/argo-workflows/issues/10178#issuecomment-2068596407 / #12965 . This is technically a regression from 3.2.
When using a semaphore, mutex, or parallelism, if your Workflow cannot start due to waiting for a lock, any PDB on it will not be created.
PDB should be created regardless of usage of semaphore or mutex
### Version
v3.5.6, latest
### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
1. Run this Workflow twice in quick succession (e.g. submit then immediately resubmit)
```YAML
metadata:
generateName: synchronization-wf-level-
spec:
podDisruptionBudget:
minAvailable: 1
synchronization:
mutex:
name: workflow
entrypoint: whalesay
templates:
- name: whalesay
container:
image: docker/whalesay:latest
command:
- sleep
args:
- "30"
```
2. Check for PDBs:
```sh
$ kubectl get pdb
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
synchronization-wf-level-6zqm6 1 N/A 0 22s
```
Only 1 PDB, not 2. The second never gets created
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Reproduce the issue by submitting the provided mutex workflow twice quickly, then inspect the workflow controller behavior and Kubernetes PDBs while the second Workflow waits for the lock. Trace the synchronization and pod disruption budget handling, using the supplied controller log command as an entry point. Done means both Workflows have their expected PDBs, including the one waiting on the mutex.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100