argoproj / argoproj/argo-workflows

Provide a timeout setting that can cancel a workflow that is queued but not yet started

Open
#6,426 2 comments 3 reactions 0 assignees View on GitHub
area/spec
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary

We want to set a timeout that automatically cancels workflows that have been stuck in a queued state. This can happen when a [synchronization](https://argoproj.github.io/argo-workflows/synchronization/) is configured or the global job limit has been exceeded.

# Use Cases

We use Argo Workflows as a task engine for a user-facing process. We set `activeDeadlineSeconds` to stop jobs that got stuck and take unexpectedly long. But sometimes the workflows are not scheduled due to an exceeded total workflow concurrency limit. In this case, we would like to eagerly cancel the scheduled job after a certain timeout to reduce the load of the system and shorten the overall work queue.

However, we didn't found a setting in the docs or while browsing the code.`activeDeadlineSeconds` limits the time in the `RUNNING` state (i.e. Kubernetes feature), while `timeout` limits the time in `PENDING + RUNNING` state. But not-yet-scheduled workflows have neither a proper state nor a start-time.

Is this already supported, or do you have an idea how this could be implemented? We could implement this kind of watchdog in our application that actively cancel stale workflows via the API, but it would be nicer as a native Argo Workflows feature.

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how Argo Workflows represents queued workflows, synchronization limits, global workflow concurrency, and activeDeadlineSeconds. Define the behavior for a native timeout that covers workflows before and after scheduling, including when cancellation should occur and how it should be observable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.