apache / apache/airflow

Improve ExternalTaskSensor execution date pattern

Open
#10,883 8 comments 4 reactions 0 assignees View on GitHub
good first issue kind:feature provider:standard
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

Currently in the `ExternalTaskSensor` there are two ways two set the execution date of a DAG/task the sensor is waiting on:
- Provide the `execution_delta` which simply adds/subtracts a time value to/from the `execution_date` of the sensor itself
- Provide the `execution_date_fn` which is a function that takes in just the `execution_date` and returns a manipulated one.

I believe that the correct way to go about handling the `execution_date` for the `ExternalTaskSensor` should be analogous to the way we handle it for other operators, such as the `ExternalTaskMarker`, or more relevantly, the `TriggerDagRunOperator`, which is an operator usually used in conjunction with the `ExternalTaskSensor`.

Specifically, we should remove the `execution_delta` parameter and replace it with a templated `execution_date` parameter. This not only covers the existing `execution_delta` use case through e.g. `{{ execution_date - macros.timedelta(days=1) }}`, but also provides more flexibility by enabling the benefits of templated parameters in general, such as pulling from xcoms, reading from DAG/operator params, using macros, etc. It also establishes the pattern that we see in the previously mentioned operators.

The proposed solution is also sufficient enough to remove the `execution_date_fn` because we can register custom macros to manipulate the execution date the same way as an `execution_date_fn` would. But I do realize we may want to keep it for backwards compatibility reasons.

First time contributor here so happy to implement/help with this to get some airflow OSS chops.

Contributor guide

Open the contributing guide

Research direction

Start with the ExternalTaskSensor implementation and inspect how execution_delta and execution_date_fn currently determine the target date. Compare that behavior with ExternalTaskMarker and TriggerDagRunOperator, then define completion as a templated execution_date supporting the stated use cases while deciding whether execution_date_fn remains for compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.