Decouple workload-token lifetime from [scheduler] task_queued_timeout
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
3.2.2
### What happened
The two-token mechanism introduced in #60108 (fixing #53713) hard-couples the workload-scoped JWT's lifetime to `[scheduler] task_queued_timeout` (see `airflow-core/docs/security/jwt_token_authentication.rst`, "Token scopes (Execution API)": *"Lifetime equals `[scheduler] task_queued_timeout`... so tuning `task_queued_timeout` also widens the window a task can wait in a backed-up queue before its workload token expires."*).
This means `task_queued_timeout` now serves two unrelated purposes at once:
1. How long a workload JWT stays valid while a task waits in an executor queue.
2. How long the scheduler waits before declaring a `queued` task orphaned/stuck and failing+retrying it (the queued-task-timeout reaper in `airflow-core/src/airflow/jobs/scheduler_job_runner.py`, around the `check_workers`/queued-timeout logic).
An operator who legitimately needs a long workload-token lifetime — e.g. to survive real queue backlog during a cron-wave burst, or to tolerate their own scheduler restarting mid-dispatch — is forced to also raise the orphaned-queued-task detection timeout to the same large value. We had to set ours to 36000s (10h) for exactly this reason.
The cost: a task that gets stuck in `queued` state for an *unrelated* reason (e.g. the scheduler gets OOMKilled in the narrow window between flipping a TI to `queued` in the DB and actually dispatching a pod/worker for it — no workload token has even been issued yet in that case) has no separate, shorter safety net. It can sit invisible and un-retried for up to 10 hours, blocking downstream tasks and the whole DAG run, because the only detection mechanism (`task_queued_timeout`) has been deliberately widened for an unrelated concern.
We checked #53713's thread: a maintainer (@amoghrajesh) already flagged this exact coupling and suggested "increase `jwt_expiration_time`" as a workaround, but that config (`[execution_api] jwt_expiration_time`) only governs the *post-`/run`* execution token — it does not affect the workload token's lifetime, so it doesn't help here. Searched for an existing issue proposing to decouple these two configs and didn't find one.
### What you think should happen instead
Add a separate, independently-tunable config for workload-token TTL (e.g. `[execution_api] workload_token_expiration_time`), decoupled from `[scheduler] task_queued_timeout`, so operators can tune:
- how long a task can legitimately wait in a backed-up queue before its dispatch token expires, and
- how quickly the scheduler detects and recovers a genuinely orphaned `queued` task
...independently of each other.
### How to reproduce
1. Set `[scheduler] task_queued_timeout` high (e.g. 36000s) to tolerate legitimate queue backlog / avoid workload-token expiry under load.
2. Have the scheduler crash/restart in the window after a TI is marked `queued` in the DB but before a pod/worker is actually dispatched for it (e.g. via OOMKill during a burst of concurrent dispatches).
3. Observe the TI remains `queued` with no pod, no heartbeat, and is not reaped until `task_queued_timeout` elapses — up to 10h in our config.
### Operating System
N/A (config/design issue, not OS-specific)
### Versions of Apache Airflow Providers
N/A
### Deployment
Other Docker-based deployment
### Deployment details
KubernetesExecutor + CeleryExecutor (multi-executor), Helm chart 1.21.0, self-hosted on EKS.
### Anything else
Related: #53713, #60108
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow-core/docs/security/jwt_token_authentication.rst and the queued-timeout logic in airflow-core/src/airflow/jobs/scheduler_job_runner.py. Trace how [scheduler] task_queued_timeout currently affects both workload-token lifetime and orphaned queued-task handling, then review the [execution_api] configuration described in the issue. Done means workload-token TTL and queued-task reaping can be tuned independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, python
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100