argoproj / argoproj/argo-workflows
Provide a timeout setting that can cancel a workflow that is queued but not yet started
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
# Summary
We want to set a timeout that automatically cancels workflows that have been stuck in a queued state. This can happen when a [synchronization](https://argoproj.github.io/argo-workflows/synchronization/) is configured or the global job limit has been exceeded.
# Use Cases
We use Argo Workflows as a task engine for a user-facing process. We set `activeDeadlineSeconds` to stop jobs that got stuck and take unexpectedly long. But sometimes the workflows are not scheduled due to an exceeded total workflow concurrency limit. In this case, we would like to eagerly cancel the scheduled job after a certain timeout to reduce the load of the system and shorten the overall work queue.
However, we didn't found a setting in the docs or while browsing the code.`activeDeadlineSeconds` limits the time in the `RUNNING` state (i.e. Kubernetes feature), while `timeout` limits the time in `PENDING + RUNNING` state. But not-yet-scheduled workflows have neither a proper state nor a start-time.
Is this already supported, or do you have an idea how this could be implemented? We could implement this kind of watchdog in our application that actively cancel stale workflows via the API, but it would be nicer as a native Argo Workflows feature.
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start by reviewing how Argo Workflows represents queued workflows, synchronization limits, global workflow concurrency, and activeDeadlineSeconds. Define the behavior for a native timeout that covers workflows before and after scheduling, including when cancellation should occur and how it should be observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100