Improve ExternalTaskSensor execution date pattern
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
Currently in the `ExternalTaskSensor` there are two ways two set the execution date of a DAG/task the sensor is waiting on:
- Provide the `execution_delta` which simply adds/subtracts a time value to/from the `execution_date` of the sensor itself
- Provide the `execution_date_fn` which is a function that takes in just the `execution_date` and returns a manipulated one.
I believe that the correct way to go about handling the `execution_date` for the `ExternalTaskSensor` should be analogous to the way we handle it for other operators, such as the `ExternalTaskMarker`, or more relevantly, the `TriggerDagRunOperator`, which is an operator usually used in conjunction with the `ExternalTaskSensor`.
Specifically, we should remove the `execution_delta` parameter and replace it with a templated `execution_date` parameter. This not only covers the existing `execution_delta` use case through e.g. `{{ execution_date - macros.timedelta(days=1) }}`, but also provides more flexibility by enabling the benefits of templated parameters in general, such as pulling from xcoms, reading from DAG/operator params, using macros, etc. It also establishes the pattern that we see in the previously mentioned operators.
The proposed solution is also sufficient enough to remove the `execution_date_fn` because we can register custom macros to manipulate the execution date the same way as an `execution_date_fn` would. But I do realize we may want to keep it for backwards compatibility reasons.
First time contributor here so happy to implement/help with this to get some airflow OSS chops.
Contributor guide
Research direction
Start with the ExternalTaskSensor implementation and inspect how execution_delta and execution_date_fn currently determine the target date. Compare that behavior with ExternalTaskMarker and TriggerDagRunOperator, then define completion as a templated execution_date supporting the stated use cases while deciding whether execution_date_fn remains for compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100