apache / apache/airflow

Decouple workload-token lifetime from [scheduler] task_queued_timeout

Open
#72,469 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

3.2.2

### What happened

The two-token mechanism introduced in #60108 (fixing #53713) hard-couples the workload-scoped JWT's lifetime to `[scheduler] task_queued_timeout` (see `airflow-core/docs/security/jwt_token_authentication.rst`, "Token scopes (Execution API)": *"Lifetime equals `[scheduler] task_queued_timeout`... so tuning `task_queued_timeout` also widens the window a task can wait in a backed-up queue before its workload token expires."*).

This means `task_queued_timeout` now serves two unrelated purposes at once:
1. How long a workload JWT stays valid while a task waits in an executor queue.
2. How long the scheduler waits before declaring a `queued` task orphaned/stuck and failing+retrying it (the queued-task-timeout reaper in `airflow-core/src/airflow/jobs/scheduler_job_runner.py`, around the `check_workers`/queued-timeout logic).

An operator who legitimately needs a long workload-token lifetime — e.g. to survive real queue backlog during a cron-wave burst, or to tolerate their own scheduler restarting mid-dispatch — is forced to also raise the orphaned-queued-task detection timeout to the same large value. We had to set ours to 36000s (10h) for exactly this reason.

The cost: a task that gets stuck in `queued` state for an *unrelated* reason (e.g. the scheduler gets OOMKilled in the narrow window between flipping a TI to `queued` in the DB and actually dispatching a pod/worker for it — no workload token has even been issued yet in that case) has no separate, shorter safety net. It can sit invisible and un-retried for up to 10 hours, blocking downstream tasks and the whole DAG run, because the only detection mechanism (`task_queued_timeout`) has been deliberately widened for an unrelated concern.

We checked #53713's thread: a maintainer (@amoghrajesh) already flagged this exact coupling and suggested "increase `jwt_expiration_time`" as a workaround, but that config (`[execution_api] jwt_expiration_time`) only governs the *post-`/run`* execution token — it does not affect the workload token's lifetime, so it doesn't help here. Searched for an existing issue proposing to decouple these two configs and didn't find one.

### What you think should happen instead

Add a separate, independently-tunable config for workload-token TTL (e.g. `[execution_api] workload_token_expiration_time`), decoupled from `[scheduler] task_queued_timeout`, so operators can tune:
- how long a task can legitimately wait in a backed-up queue before its dispatch token expires, and
- how quickly the scheduler detects and recovers a genuinely orphaned `queued` task

...independently of each other.

### How to reproduce

1. Set `[scheduler] task_queued_timeout` high (e.g. 36000s) to tolerate legitimate queue backlog / avoid workload-token expiry under load.
2. Have the scheduler crash/restart in the window after a TI is marked `queued` in the DB but before a pod/worker is actually dispatched for it (e.g. via OOMKill during a burst of concurrent dispatches).
3. Observe the TI remains `queued` with no pod, no heartbeat, and is not reaped until `task_queued_timeout` elapses — up to 10h in our config.

### Operating System

N/A (config/design issue, not OS-specific)

### Versions of Apache Airflow Providers

N/A

### Deployment

Other Docker-based deployment

### Deployment details

KubernetesExecutor + CeleryExecutor (multi-executor), Helm chart 1.21.0, self-hosted on EKS.

### Anything else

Related: #53713, #60108

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow-core/docs/security/jwt_token_authentication.rst and the queued-timeout logic in airflow-core/src/airflow/jobs/scheduler_job_runner.py. Trace how [scheduler] task_queued_timeout currently affects both workload-token lifetime and orphaned queued-task handling, then review the [execution_api] configuration described in the issue. Done means workload-token TTL and queued-task reaping can be tuned independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.