envoyproxy / envoyproxy/gateway
WebSocket upgrade returns 403 (upgrade_failed) instead of 503 when the backend has no ready endpoints
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
## Description
When an HTTPRoute's backend has **zero ready endpoints**, Envoy Gateway replaces the forwarding route with a `direct_response: {status: 503}` placeholder. That placeholder route has **no `upgrade_configs`**, so the HTTP Connection Manager rejects any WebSocket `Upgrade` on that route with **`403` (`response_code_details: upgrade_failed`)** before the direct response fires.
Result: on the *same route at the same instant*, a plain request correctly gets `503`, but a WebSocket upgrade gets a misleading `403`. Clients that (correctly) treat `503` as retryable but `403` as a terminal auth failure will tear down instead of reconnecting. This makes every rolling update of a single-replica / `Recreate` backend that serves WebSockets terminate its subscribers.
## Environment
- Envoy Gateway **v1.8.1** (`gateway-helm:1.8.1`), Envoy **1.38.1**
- Plain HTTPRoute, no explicit upgrade config (relying on EG's default per-route WebSocket enablement)
## Steps to reproduce
Deterministic — no rollout timing needed:
```
# healthy backend behind a plain HTTPRoute → WS upgrade returns 101
kubectl scale deploy/ --replicas=0 # drive backend to zero ready endpoints
curl -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
-H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' -H 'Sec-WebSocket-Version: 13' \
http:/// # → 403 (upgrade_failed)
curl http:/// # → 503 (correct)
kubectl scale deploy/ --replicas=1 # → WS upgrade returns 101 again
```
## Evidence (config_dump, `dynamic_route_configs`)
Backend at **0 ready endpoints** — route is a direct 503 with no upgrade support:
```json
{
"name": "httproute/demo/ws-echo/rule/0/match/0/*",
"match": { "prefix": "/" },
"direct_response": { "status": 503 }
}
```
Backend at **1 ready endpoint** — forwarding route with the WebSocket upgrade:
```json
{
"name": "httproute/demo/ws-echo/rule/0/match/0/*",
"match": { "prefix": "/" },
"route": {
"cluster": "httproute/demo/ws-echo/rule/0",
"upgrade_configs": [ { "upgrade_type": "websocket" } ]
}
}
```
Gateway access log during the window (`%RESPONSE_CODE_DETAILS%`): `403 upgrade_failed` and `503 direct_response` on the same route, `upstream_host: null` (local reply).
## Expected
A WebSocket upgrade to a route whose backend is temporarily unavailable should surface a **retryable `503`**, consistent with the non-upgrade path — not a `403`. Either preserve `upgrade_configs` on the no-endpoint placeholder route, or don't substitute a `direct_response` route at all (let the cluster return `503 no_healthy_upstream`, which keeps the real route + its upgrade config intact).
## Notes
Related but distinct: #8889 (periodic xDS republish draining established WS connections) and #8578 (requestBuffer disabling WS upgrades). This is a separate cause — the no-ready-endpoints route substitution dropping `upgrade_configs`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Envoy Gateway's HTTPRoute translation for a backend with zero ready endpoints, using the config_dump examples as the expected route shapes. Compare the no-endpoint direct_response route with the healthy route's upgrade_configs, then reproduce with the provided kubectl scale and curl commands. Done means a WebSocket upgrade during the no-endpoint window returns retryable 503 rather than 403 upgrade_failed, while the healthy path still returns 101.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100