apache / apache/airflow

Scheduler crash when API server returns 301 or redirection through proxy

Open
#49,099 3 comments 0 reactions 0 assignees View on GitHub
affected_version:3.0 affected_version:main_branch area:core area:scheduler kind:bug priority:low Stale Bug Report
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

main (development)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

`get_json_error` checks for status to be of the series 4xx or 5xx and if not returns None. In case of redirections like 301 through corporate proxy it returns 3xx status code. Thus `get_json_error` returns None and `response.raise_for_status` raises HTTPStatusError which is not pickleable and scheduler crashes similar to #47873

https://github.com/apache/airflow/blob/9b1a507483a86aa87e4f70621dd9af8afb1f1f21/task-sdk/src/airflow/sdk/api/client.py#L114-L123

### What you think should happen instead?

_No response_

### How to reproduce

1. Run scheduler in a proxy environment where 3xx might be returned.
2. Create a dagrun.
3. Scheduler crashes with `HTTPStatusError` which is not pickleable

### Operating System

Ubuntu 20.04

### Versions of Apache Airflow Providers

_No response_

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in task-sdk/src/airflow/sdk/api/client.py around get_json_error and response.raise_for_status, then reproduce the scheduler failure with a 301 response through a proxy. Trace how the response is handled and verify that the scheduler no longer crashes when the API server returns a redirect.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.