argoproj / argoproj/argo-workflows

Lock message misleading when workflow is not at front of queue

Open
#14,080 1 comment 0 reactions 0 assignees View on GitHub
area/mutex-semaphore
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [ ] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

When running a workflow with multiple synchronization locks, where one is a mutex and one is a semaphore, you can get into a case where the shown Pending message is misleading/confusing.

Since a workflow must wait for all locks to be available, but the queue is processed in order, if you have say 5 jobs in the queue that all rely on the same mutex, and then 5 more jobs behind them that don't, but they all rely on the same semaphore, the jobs that had a different mutex will not run until all of the jobs relying on the same mutex finish.

This is not the issue, this is expected and fully explained in the docs. However, the message the system returns for those not relying on the mutex is `fmt.Sprintf("Waiting for %s lock. Lock status: %d/%d", s.name, s.limit-len(s.lockHolder), s.limit)` from https://github.com/argoproj/argo-workflows/blob/v3.6.2/workflow/sync/semaphore.go#L176.

What is misleading is the case where there is plenty of semaphore room, but you are violating your position in queue. You end up with a pending message that says waiting for lock but shows you plenty of lock room, but what it really should say is "Waiting for position in queue" or something.

This would help troubleshooting situations as to why not a lot of jobs are running even though the semaphore has lots of room.

### Version(s)

v3.6.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
Example workflow level synchronization config, where the semaphore is set to say 200 and the mutex has 5 jobs with the same key in front of a job with a different key

synchronization:
semaphore:
configMapKeyRef:
name: my_config_map
key: lots_of_jobs
namespace: default
mutex:
name: my_uuid
namespace: default
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start in workflow/sync/semaphore.go at the semaphore message around line 176 and review the synchronization queue handling. Reproduce the mutex-and-semaphore configuration described in the issue, then verify that a workflow blocked by queue position receives a message that distinguishes queue waiting from available lock capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.