apache / apache/airflow

HttpSensor deferrable mode ignores response_error_codes_allowlist

Open
#69,903 0 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug provider:http
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### What happened and how to reproduce it?

`HttpSensor` supports `response_error_codes_allowlist` so users can treat selected HTTP errors as "keep poking" instead of failing the task. In the synchronous path this is honored by checking `self.response_error_codes_allowlist` in `HttpSensor.poke()`.

However, in deferrable mode the behavior appears to diverge after the task is deferred:

- `providers/http/src/airflow/providers/http/sensors/http.py` stores the configured allowlist and uses it in `poke()`.
- `HttpSensor.execute()` creates an `HttpSensorTrigger`, but does not pass `response_error_codes_allowlist` to the trigger.
- `providers/http/src/airflow/providers/http/triggers/http.py` has `HttpSensorTrigger.run()` retry only when the exception string starts with `"404"`.

A DAG using something like this can behave differently depending on whether deferrable mode is enabled:

```python
HttpSensor(
task_id="wait_for_api",
endpoint="/health",
http_conn_id="my_api",
response_error_codes_allowlist=["404", "503"],
deferrable=True,
)
```

Expected behavior: `503` should be treated the same way as in non-deferrable mode, i.e. the sensor should keep waiting according to `poke_interval`.

Current behavior: the initial worker-side `poke()` can honor `503`, but after deferral the trigger only handles `404`. For other allowlisted statuses such as `503`, the trigger does not sleep via the allowlist path and does not preserve the documented sensor behavior.

### What you think should happen instead?

`HttpSensorTrigger` should accept and serialize the configured `response_error_codes_allowlist`, and `HttpSensor.execute()` should pass the value when creating the trigger.

The trigger should use that allowlist instead of hard-coding `404`, so deferrable and non-deferrable `HttpSensor` behavior are consistent.

A unit test should cover a deferrable `HttpSensor` with a custom allowlist such as `["404", "503"]`.

Contributor guide

Open the contributing guide

Research direction

Start in providers/http/src/airflow/providers/http/sensors/http.py, tracing HttpSensor.execute() and its deferral path, then compare it with HttpSensorTrigger.run() in providers/http/src/airflow/providers/http/triggers/http.py. Add or update the relevant unit coverage for a deferrable sensor using ["404", "503"], and confirm that 503 follows the same waiting behavior as the synchronous path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.