apache / apache/airflow

Move dag-level callbacks to worker

Open
#44,354 14 comments 1 reaction 0 assignees View on GitHub
area:task-execution-interface-aip72
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

We will move callbacks to workers. TBD on “how”.

Options:

1. Moving callbacks to workers as a separate activity.
2. Running callbacks as teardown task.
3. Deprecate (not remove) concept of callback and pushing users to use teardown. If we do this and deprecate/not remove this option will happen with one of the above two options.
4. Leave them in dag-processor

I am leaning towards (2) — and (3) is my next preference. This is because imo callbacks and teardowns are sort of redundant concepts, and if we keep them as is, we will have an explosion of callbacks, we already have: `on_{success,failure,skipped,retry,execute}_callback` .

Contributor guide

Open the contributing guide

Research direction

The issue names DAG-level callbacks, workers, the dag-processor, and teardown tasks, but provides no files or tests. First map those entry points and review the unresolved options; done requires an agreed design for callback handling and its implementation and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.