Cover clearing task state based on dag run clear
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Why?
#72100 added discard-by-default behavior for `task_state_store` entries when a task instance
is cleared via the `POST /dags/{dag_id}/clearTaskInstances` endpoint — checkpoints are
discarded unless `keep_task_state` is set, so a cleared task starts over instead of silently
resuming from a stale checkpoint after the underlying code/data was fixed.
That fix only applies to that one endpoint. Every other way of clearing a task instance still
keeps task state unconditionally, because each of them calls `clear_task_instances()` directly
instead of going through the new discard logic:
- Clearing an entire Dag run (the "Clear Run" dialog / API) —
`perform_clear_dag_run` -> `dag.clear()` -> `clear_task_instances()`.
- Marking a task as failed/success, which clears downstream tasks as a side effect.
- `airflow dags clear` / `airflow tasks clear` (core CLI) —
`cli/commands/dag_command.py` -> `clear_task_instances()` directly.
### Follow-up work
Thread `keep_task_state` (or an equivalent opt-out) through `clear_task_instances()` itself, so
every caller (REST endpoint, Dag-run clear, CLI, mark-as-success/failed) converges on the same
discard-by-default behavior. This likely needs:
- A `keep_task_state` parameter on `clear_task_instances()`.
- Plumbing it through `dag.clear()`, `perform_clear_dag_run`, and the core CLI's
`airflow dags clear` / `airflow tasks clear`.
- Deciding the right default/UX for callers that have no request body to carry the flag today
(e.g. CLI defaults, mark-as-success/failed).
### Acceptance criteria
- Clearing a Dag run discards `task_state_store` entries by default, matching the task-instance
clear endpoint, with an equivalent opt-out to keep state.
- `airflow dags clear` / `airflow tasks clear` behave consistently with the REST endpoints.
- Docs (`resumable-tasks.rst`) no longer need to caveat which clear surface actually discards.
Contributor guide
Assessment
This issue has not been assessed yet.