Tasks returning from deferred should be prioritized
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Tasks that are coming out of deferred should be prioritized so that they run first.
This is the query to queue `scheduled` tasks (the state that tasks go into after comming out of deferral).
```
query = (
select(TI)
.with_hint(TI, "USE INDEX (ti_state)", dialect_name="mysql")
.join(TI.dag_run)
.where(DR.state == DagRunState.RUNNING)
.join(TI.dag_model)
.where(~DM.is_paused)
.where(TI.state == TaskInstanceState.SCHEDULED)
.where(DM.bundle_name.is_not(None))
.options(selectinload(TI.dag_model))
.order_by(-TI.priority_weight, DR.logical_date, TI.map_index)
)
```
One option would be to increase their priority_weight.
### Use case/motivation
When a task leaves deferred, it (in most cases) needs to go back to scheduled to get to a worker for execute_complete. In cases where the task's pool is full when it finishes, this can lead to long delays between when a task's main goal is finished and it actually completing. For instance, in KPO, this can lead to the pod not being around anymore by the time the execute_complete tries to read the final logs.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the shown query that queues scheduled tasks, then trace how tasks leaving deferred enter the scheduled state and how priority_weight affects ordering. Compare the proposed priority-weight change with the scheduler behavior described in the issue. Done means tasks returning from deferred are selected ahead of other scheduled tasks, reducing delays before execute_complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100