Move dag-level callbacks to worker
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
We will move callbacks to workers. TBD on “how”.
Options:
1. Moving callbacks to workers as a separate activity.
2. Running callbacks as teardown task.
3. Deprecate (not remove) concept of callback and pushing users to use teardown. If we do this and deprecate/not remove this option will happen with one of the above two options.
4. Leave them in dag-processor
I am leaning towards (2) — and (3) is my next preference. This is because imo callbacks and teardowns are sort of redundant concepts, and if we keep them as is, we will have an explosion of callbacks, we already have: `on_{success,failure,skipped,retry,execute}_callback` .
Contributor guide
Research direction
The issue names DAG-level callbacks, workers, the dag-processor, and teardown tasks, but provides no files or tests. First map those entry points and review the unresolved options; done requires an agreed design for callback handling and its implementation and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100