ExternalTaskSensor: An option to use the data_interval variables rather than execution_date
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Since `execution_date` is deprecated, it makes sense that this sensor or a new version of it (for backward compatibility) uses the `data_interval` variables.
### Use case/motivation
I want to make sure that the data interval this DAG is about to operate on is covered by another DAG. For example, data was loaded up to this DAG's `data_interval_end`.
I believe the sensor itself is simple to write. It just uses `data_interval_end` rather than `execution_date` and it doesn't look for one specific task, any `external.data_interval_end` > `self.data_interval_end` is acceptable.
This is the equivalent of `ExternalTaskSensor` with `execution_delta` set to `self.interval_start - closest_external_interval.interval_start` which can be calculated manually but will fail later if the external DAG schedule is changed.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the ExternalTaskSensor entry point and read how execution_date and execution_delta currently determine external coverage. Compare that behavior with data_interval_start and data_interval_end, then define whether the option applies to a sensor or a backward-compatible version; done means a DAG can verify that an external interval covers its own interval without relying on a fixed schedule offset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100