apache / apache/airflow

ExternalTaskSensor does not check task or task-group existence on Airflow 3

Open
#72,514 3 comments 0 reactions 1 assignee Claimed by @ignacioparicio View on GitHub
area:providers kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

`main` at `b3b62fa8cf`; the problem affects Airflow 3.

### What happened and how to reproduce it?

On Airflow 3, `ExternalTaskSensor(check_existence=True)` cannot check whether an external task or task group exists.

For example, if `external_task_id` contains a typo, the sensor keeps waiting until its timeout instead of reporting that the task does not exist. The same happens with a missing `external_task_group_id`.

This can be reproduced by creating an `ExternalTaskSensor` with:

- `check_existence=True`
- a valid `external_dag_id`
- an `external_task_id` that does not exist

The sensor waits instead of raising `ExternalTaskNotFoundError`.

Airflow 3 workers cannot read the metadata database or load Dag files directly. They ask the Execution API for task states instead. A response containing no task instances is ambiguous: the task might not exist, or the expected Dag run might not have been created yet.

The related Dag-level fix checks only whether the external Dag exists and warns that individual tasks and task groups were not checked.

### What do you think should happen instead?

`check_existence=True` should report a missing external task or task group without mistaking a missing Dag run for a missing task.

The check also needs to use the Dag version associated with the run being awaited. If that run does not exist yet, the sensor should continue waiting rather than assume that the task is missing.

The expected behavior should work both when the sensor waits itself and when it hands the wait to the triggerer.

### Apache Airflow Provider(s)

standard

### Versions of Apache Airflow Providers

`main` (`apache-airflow-providers-standard==1.17.0` at the time of reporting)

### Official Helm Chart version

Not Applicable

### Anything else?

#67832 explored separate task and task-group existence endpoints. It was closed after the problem of choosing the correct Dag version was raised.

Acceptance criteria:

- Missing external tasks and task groups can be distinguished from missing Dag runs.
- The relevant Dag version is used.
- Both ways of running the sensor are covered by tests.

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md).

---
Drafted-by: Codex (GPT-5); reviewed by @ignacioparicio before posting

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.