apache / apache/airflow

Deferred sensors with soft_fail=True lack parity skip on non-timeout failures

Open
#71,255 2 comments 0 reactions 0 assignees View on GitHub
area:async-operators area:core-operators kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

2.11.2

### What happened and how to reproduce it?

When running sensors with `deferred=False`, mode `reschedule` or `poke`, and `soft_fail=True`, unexpected errors will actually fail, instead of skip, but timeout errors will skip.

When doing the same, but with `soft_fail=True`, and `deferred=True`, unexpected errors will cause the task to skip. This means we cannot distinguish and alert if the sensor is misconfigured vs. timeout.

### What you think should happen instead?

Deferred sensors should have parity with non-deferred sensors, and if `soft_fail=True`, only skip on timeout-ish related errors

### Operating System

linux

### Deployment

Other Docker-based deployment

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

n/a, issue is in `main`/`master`

### Official Helm Chart version

Not Applicable

### Kubernetes Version

n/z

### Helm Chart configuration

n/a

### Docker Image customizations

n/a

### Anything else?

n/a

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by comparing deferred sensor exception handling with the existing non-deferred reschedule and poke behavior for soft_fail=True. Confirm the timeout-related and unexpected-error paths, then add regression coverage showing that deferred sensors skip only timeout-ish failures and propagate unexpected errors like non-deferred sensors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.