envoyproxy / envoyproxy/gateway

WebSocket upgrade returns 403 (upgrade_failed) instead of 503 when the backend has no ready endpoints

Open
#9,452 1 comment 0 reactions 0 assignees View on GitHub
stale triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

## Description

When an HTTPRoute's backend has **zero ready endpoints**, Envoy Gateway replaces the forwarding route with a `direct_response: {status: 503}` placeholder. That placeholder route has **no `upgrade_configs`**, so the HTTP Connection Manager rejects any WebSocket `Upgrade` on that route with **`403` (`response_code_details: upgrade_failed`)** before the direct response fires.

Result: on the *same route at the same instant*, a plain request correctly gets `503`, but a WebSocket upgrade gets a misleading `403`. Clients that (correctly) treat `503` as retryable but `403` as a terminal auth failure will tear down instead of reconnecting. This makes every rolling update of a single-replica / `Recreate` backend that serves WebSockets terminate its subscribers.

## Environment

- Envoy Gateway **v1.8.1** (`gateway-helm:1.8.1`), Envoy **1.38.1**
- Plain HTTPRoute, no explicit upgrade config (relying on EG's default per-route WebSocket enablement)

## Steps to reproduce

Deterministic — no rollout timing needed:

```
# healthy backend behind a plain HTTPRoute → WS upgrade returns 101
kubectl scale deploy/ --replicas=0 # drive backend to zero ready endpoints

curl -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
-H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' -H 'Sec-WebSocket-Version: 13' \
http:/// # → 403 (upgrade_failed)
curl http:/// # → 503 (correct)

kubectl scale deploy/ --replicas=1 # → WS upgrade returns 101 again
```

## Evidence (config_dump, `dynamic_route_configs`)

Backend at **0 ready endpoints** — route is a direct 503 with no upgrade support:
```json
{
"name": "httproute/demo/ws-echo/rule/0/match/0/*",
"match": { "prefix": "/" },
"direct_response": { "status": 503 }
}
```

Backend at **1 ready endpoint** — forwarding route with the WebSocket upgrade:
```json
{
"name": "httproute/demo/ws-echo/rule/0/match/0/*",
"match": { "prefix": "/" },
"route": {
"cluster": "httproute/demo/ws-echo/rule/0",
"upgrade_configs": [ { "upgrade_type": "websocket" } ]
}
}
```

Gateway access log during the window (`%RESPONSE_CODE_DETAILS%`): `403 upgrade_failed` and `503 direct_response` on the same route, `upstream_host: null` (local reply).

## Expected

A WebSocket upgrade to a route whose backend is temporarily unavailable should surface a **retryable `503`**, consistent with the non-upgrade path — not a `403`. Either preserve `upgrade_configs` on the no-endpoint placeholder route, or don't substitute a `direct_response` route at all (let the cluster return `503 no_healthy_upstream`, which keeps the real route + its upgrade config intact).

## Notes

Related but distinct: #8889 (periodic xDS republish draining established WS connections) and #8578 (requestBuffer disabling WS upgrades). This is a separate cause — the no-ready-endpoints route substitution dropping `upgrade_configs`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Envoy Gateway's HTTPRoute translation for a backend with zero ready endpoints, using the config_dump examples as the expected route shapes. Compare the no-endpoint direct_response route with the healthy route's upgrade_configs, then reproduce with the provided kubectl scale and curl commands. Done means a WebSocket upgrade during the no-endpoint window returns retryable 503 rather than 403 upgrade_failed, while the healthy path still returns 101.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend-api-design, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.