Reset executor_config when clearing a task?
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
Currently if you clear a task, it won't clear the executor config. The executor config is pickled and added to the ti row at time of dag run creation by the dag processor. And when you clear a TI the state us updated but the executor_config remains unchanged.
In order to have the executor_config update properly we may have to have TI clear function as a dag processor callback, alternatively we could have clear in effect delete the TI row and then ... well what would happen? i think the scheduler would recreate it? but it doesn't even have access to the dag so how does it get the executor config?
@ash @jedcunningham
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by tracing the task-instance clear function, the dag processor callback path, and how the scheduler recreates task instances, as described in the issue. Determine how clearing should refresh the pickled executor_config, then verify that a cleared task instance receives the correct configuration on its next DAG run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100