apache / apache/airflow

ExternalTaskSensor: An option to use the data_interval variables rather than execution_date

Open
#26,429 5 comments 1 reaction 0 assignees View on GitHub
good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Since `execution_date` is deprecated, it makes sense that this sensor or a new version of it (for backward compatibility) uses the `data_interval` variables.

### Use case/motivation

I want to make sure that the data interval this DAG is about to operate on is covered by another DAG. For example, data was loaded up to this DAG's `data_interval_end`.

I believe the sensor itself is simple to write. It just uses `data_interval_end` rather than `execution_date` and it doesn't look for one specific task, any `external.data_interval_end` > `self.data_interval_end` is acceptable.

This is the equivalent of `ExternalTaskSensor` with `execution_delta` set to `self.interval_start - closest_external_interval.interval_start` which can be calculated manually but will fail later if the external DAG schedule is changed.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the ExternalTaskSensor entry point and read how execution_date and execution_delta currently determine external coverage. Compare that behavior with data_interval_start and data_interval_end, then define whether the option applies to a sensor or a backward-compatible version; done means a DAG can verify that an external interval covers its own interval without relying on a fixed schedule offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.