apache / apache/airflow

Tasks returning from deferred should be prioritized

Open
#57,210 6 comments 3 reactions 0 assignees View on GitHub
area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Tasks that are coming out of deferred should be prioritized so that they run first.

This is the query to queue `scheduled` tasks (the state that tasks go into after comming out of deferral).

```
query = (
select(TI)
.with_hint(TI, "USE INDEX (ti_state)", dialect_name="mysql")
.join(TI.dag_run)
.where(DR.state == DagRunState.RUNNING)
.join(TI.dag_model)
.where(~DM.is_paused)
.where(TI.state == TaskInstanceState.SCHEDULED)
.where(DM.bundle_name.is_not(None))
.options(selectinload(TI.dag_model))
.order_by(-TI.priority_weight, DR.logical_date, TI.map_index)
)
```

One option would be to increase their priority_weight.

### Use case/motivation

When a task leaves deferred, it (in most cases) needs to go back to scheduled to get to a worker for execute_complete. In cases where the task's pool is full when it finishes, this can lead to long delays between when a task's main goal is finished and it actually completing. For instance, in KPO, this can lead to the pod not being around anymore by the time the execute_complete tries to read the final logs.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the shown query that queues scheduled tasks, then trace how tasks leaving deferred enter the scheduled state and how priority_weight affects ordering. Compare the proposed priority-weight change with the scheduler behavior described in the issue. Done means tasks returning from deferred are selected ahead of other scheduled tasks, reducing delays before execute_complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.