HttpSensor deferrable mode ignores response_error_codes_allowlist
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### What happened and how to reproduce it?
`HttpSensor` supports `response_error_codes_allowlist` so users can treat selected HTTP errors as "keep poking" instead of failing the task. In the synchronous path this is honored by checking `self.response_error_codes_allowlist` in `HttpSensor.poke()`.
However, in deferrable mode the behavior appears to diverge after the task is deferred:
- `providers/http/src/airflow/providers/http/sensors/http.py` stores the configured allowlist and uses it in `poke()`.
- `HttpSensor.execute()` creates an `HttpSensorTrigger`, but does not pass `response_error_codes_allowlist` to the trigger.
- `providers/http/src/airflow/providers/http/triggers/http.py` has `HttpSensorTrigger.run()` retry only when the exception string starts with `"404"`.
A DAG using something like this can behave differently depending on whether deferrable mode is enabled:
```python
HttpSensor(
task_id="wait_for_api",
endpoint="/health",
http_conn_id="my_api",
response_error_codes_allowlist=["404", "503"],
deferrable=True,
)
```
Expected behavior: `503` should be treated the same way as in non-deferrable mode, i.e. the sensor should keep waiting according to `poke_interval`.
Current behavior: the initial worker-side `poke()` can honor `503`, but after deferral the trigger only handles `404`. For other allowlisted statuses such as `503`, the trigger does not sleep via the allowlist path and does not preserve the documented sensor behavior.
### What you think should happen instead?
`HttpSensorTrigger` should accept and serialize the configured `response_error_codes_allowlist`, and `HttpSensor.execute()` should pass the value when creating the trigger.
The trigger should use that allowlist instead of hard-coding `404`, so deferrable and non-deferrable `HttpSensor` behavior are consistent.
A unit test should cover a deferrable `HttpSensor` with a custom allowlist such as `["404", "503"]`.
Contributor guide
Research direction
Start in providers/http/src/airflow/providers/http/sensors/http.py, tracing HttpSensor.execute() and its deferral path, then compare it with HttpSensorTrigger.run() in providers/http/src/airflow/providers/http/triggers/http.py. Add or update the relevant unit coverage for a deferrable sensor using ["404", "503"], and confirm that 503 follows the same waiting behavior as the synchronous path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100