apache / apache/airflow

Cover clearing task state based on dag run clear

Open
#72,929 0 comments 0 reactions 1 assignee Claimed by @amoghrajesh View on GitHub
area:core kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Why?

#72100 added discard-by-default behavior for `task_state_store` entries when a task instance
is cleared via the `POST /dags/{dag_id}/clearTaskInstances` endpoint — checkpoints are
discarded unless `keep_task_state` is set, so a cleared task starts over instead of silently
resuming from a stale checkpoint after the underlying code/data was fixed.

That fix only applies to that one endpoint. Every other way of clearing a task instance still
keeps task state unconditionally, because each of them calls `clear_task_instances()` directly
instead of going through the new discard logic:

- Clearing an entire Dag run (the "Clear Run" dialog / API) —
`perform_clear_dag_run` -> `dag.clear()` -> `clear_task_instances()`.
- Marking a task as failed/success, which clears downstream tasks as a side effect.
- `airflow dags clear` / `airflow tasks clear` (core CLI) —
`cli/commands/dag_command.py` -> `clear_task_instances()` directly.

### Follow-up work

Thread `keep_task_state` (or an equivalent opt-out) through `clear_task_instances()` itself, so
every caller (REST endpoint, Dag-run clear, CLI, mark-as-success/failed) converges on the same
discard-by-default behavior. This likely needs:

- A `keep_task_state` parameter on `clear_task_instances()`.
- Plumbing it through `dag.clear()`, `perform_clear_dag_run`, and the core CLI's
`airflow dags clear` / `airflow tasks clear`.
- Deciding the right default/UX for callers that have no request body to carry the flag today
(e.g. CLI defaults, mark-as-success/failed).

### Acceptance criteria

- Clearing a Dag run discards `task_state_store` entries by default, matching the task-instance
clear endpoint, with an equivalent opt-out to keep state.
- `airflow dags clear` / `airflow tasks clear` behave consistently with the REST endpoints.
- Docs (`resumable-tasks.rst`) no longer need to caveat which clear surface actually discards.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.