apache / apache/airflow

Tasks that are running when a DAG run reaches its timeout should optionally support `on_killed`

Open
#41,036 9 comments 1 reaction 0 assignees View on GitHub
area:scheduler good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

The current functionality for task instances that are in the running state when a DAG run reaches its timeout is controversial (see #30264). It'd be great if users could optionally allow tasks' `on_kill` method to be called at this time.

### Use case/motivation

Some users would like for externally running workloads (e.g. snowflake, emr, bigquery, etc etc) to stop executing when a DAG run times out.

### Related issues

#30264

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading the current timeout behavior and the discussion in related issue #30264, focusing on how running task instances are handled when a DAG run times out. Define the optional behavior and verify that externally running workloads can receive on_kill without changing the default behavior; the issue names no files or tests, so those must be located during investigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.