apache / apache/airflow

Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API

Open
#39,463 9 comments 0 reactions 0 assignees View on GitHub
area:core area:providers kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I'm proposing to add a new ExternalAPITaskSensor to the Apache Airflow project. This sensor leverages the Airflow REST API to monitor the status of an external DAG or task. It can be used to check the completion of another DAG or task before proceeding with the execution of the current task, providing better orchestration across DAGs.

### Use case/motivation

The new sensor provides the ability to monitor the status of external DAGs using the Airflow REST API, which is particularly useful in a distributed environment where multiple Airflow instances are running.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing existing Airflow sensors and the Airflow REST API endpoints for external DAG and task status. Define how the new sensor should identify and monitor a DAG or task across Airflow instances, and consider the test coverage needed to show that it waits for completion before allowing the current task to proceed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.