Scheduler crash when API server returns 301 or redirection through proxy
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
`get_json_error` checks for status to be of the series 4xx or 5xx and if not returns None. In case of redirections like 301 through corporate proxy it returns 3xx status code. Thus `get_json_error` returns None and `response.raise_for_status` raises HTTPStatusError which is not pickleable and scheduler crashes similar to #47873
https://github.com/apache/airflow/blob/9b1a507483a86aa87e4f70621dd9af8afb1f1f21/task-sdk/src/airflow/sdk/api/client.py#L114-L123
### What you think should happen instead?
_No response_
### How to reproduce
1. Run scheduler in a proxy environment where 3xx might be returned.
2. Create a dagrun.
3. Scheduler crashes with `HTTPStatusError` which is not pickleable
### Operating System
Ubuntu 20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in task-sdk/src/airflow/sdk/api/client.py around get_json_error and response.raise_for_status, then reproduce the scheduler failure with a 301 response through a proxy. Trace how the response is handled and verify that the scheduler no longer crashes when the API server returns a redirect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100