AppProxy worker rejects any request whose X-Forwarded-For hop count is not exactly one
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
### Describe the bug
With proxy_worker.trusted_proxies configured, the worker answers 400 to every HTTP request whose X-Forwarded-For header does not carry exactly one entry.
```python
# src/ai/backend/appproxy/worker/proxy/frontend/http/base.py
self._xff_strict = XForwardedStrict([list(trusted_proxies)]) if trusted_proxies else None
```
Wrapping the setting in another list declares exactly one trusted hop, no matter how many networks the setting holds. aiohttp_remotes then enforces the hop count before it ever checks membership:
```python
# aiohttp_remotes/utils.py, remote_ip()
if len(trusted) + 1 != len(ips): # ips = [peer] + reversed(X-Forwarded-For)
raise IncorrectIPCount(...) # -> HTTP 400
```
So len(ips) must be 2, i.e. exactly one X-Forwarded-For entry.
### Impact
- A request that arrives without X-Forwarded-For (direct call, health check) is rejected.
- A deployment with two or more proxies in front of the worker is rejected, and the setting cannot express it: adding more entries to trusted_proxies does not add hops.
- This blocks the per-circuit allowed_client_ips enforcement, which is the feature that requires trusted_proxies in the first place.
### To Reproduce
1. Set proxy_worker.trusted_proxies to the network of the proxy in front of the worker.
1. Send a request through the worker with no X-Forwarded-For header -> 400.
1. Send a request with two X-Forwarded-For entries, both inside trusted_proxies -> 400.
1. Send a request with exactly one entry -> served.
Measured on the manager, which had the identical construction until BA-7252: with trusted-proxies = ["127.0.0.1/32", "10.0.0.0/8"] and peer 127.0.0.1, every chain length other than one returned 400, regardless of whether each entry was in the trusted list. The worker has not been measured directly, but it builds the middleware the same way.
### Expected Behavior
- Requests are served regardless of how many proxy hops are in front of the worker, including none.
- The client IP used for the allowed_client_ips check is the first address that is not a trusted proxy, walking the chain inwards from the connection peer.
- A client that forges X-Forwarded-For while connecting directly is not able to change the address used for the check.
### Anything else?
- Affects the worker from 26.4.5, when proxy_worker.trusted_proxies was added.
- Unlike the manager case, the worker registers the middleware first in the list, so its execution order is correct; only the hop count is wrong.
- The coordinator already expresses the intended behavior in its Traefik ipStrategy.excluded_ips setting.
JIRA Issue: BA-7320
Contributor guide
Research direction
Start in src/ai/backend/appproxy/worker/proxy/frontend/http/base.py, then inspect aiohttp_remotes/utils.py and its remote_ip() hop-count handling. Reproduce requests with no, one, and multiple X-Forwarded-For entries using trusted_proxies. Done means valid proxy chains are accepted, the first untrusted address is used for allowed_client_ips, and direct clients cannot forge that address.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100