apache / apache/airflow

Reset executor_config when clearing a task?

Open
#26,568 15 comments 0 reactions 0 assignees View on GitHub
kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

Currently if you clear a task, it won't clear the executor config. The executor config is pickled and added to the ti row at time of dag run creation by the dag processor. And when you clear a TI the state us updated but the executor_config remains unchanged.

In order to have the executor_config update properly we may have to have TI clear function as a dag processor callback, alternatively we could have clear in effect delete the TI row and then ... well what would happen? i think the scheduler would recreate it? but it doesn't even have access to the dag so how does it get the executor config?

@ash @jedcunningham

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the task-instance clear function, the dag processor callback path, and how the scheduler recreates task instances, as described in the issue. Determine how clearing should refresh the pickled executor_config, then verify that a cleared task instance receives the correct configuration on its next DAG run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.