envoyproxy / envoyproxy/envoy

Cluster: fallback to HTTP/1.1 Websocket in auto_config when RFC 8441 ws over HTTP/2 negotiation fails (or always permit websocket over h1)

Open
#37,020 6 comments 0 reactions 0 assignees View on GitHub
area/upstream enhancement no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

*Description*:
We have the following cluster configuration to ensure that HTTP/2 is used when signaled via ALPN, while still retaining the ability to fall back to HTTP/1.1 if HTTP/2 is not in the ALPN. This simplifies our xDS in a environment with mixed backends.
```
"typed_extension_protocol_options": {
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
"common_http_protocol_options": {
"idle_timeout": "55s"
},
"auto_config": {
"http_protocol_options": {
"header_key_format": {
"proper_case_words": {
}
}
},
"http2_protocol_options": {
"max_concurrent_streams": 128,
"initial_stream_window_size": 1048576,
"allow_connect": true
}
}
}
},
```

You'll see that `allow_connect` is present, which enables support for RFC 8441 (websocket over http/2 via extended connect), which expects the backend to send `SETTINGS_ENABLE_CONNECT_PROTOCOL=1`.

We find that if the backend DOES support HTTP/2 but does NOT support RFC 8441 `SETTINGS_ENABLE_CONNECT_PROTOCOL=1` (e.g. AWS ALB) Envoy does not gracefully fall back to using HTTP/1.1 for the websocket, and the connections fail. The only recourse is to disable HTTP/2 for the cluster entirely.

Even if `allow_connect` is `false`, Envoy fails to proxy the Websocket connection if HTTP/2 is supported in ALPN.

It appears Envoy lacks logic to infer that a Websocket connection with `allow_connect: false` + ALPN `h2 http1.1`, or with `allow_connect :true` + ALPN `h2 http1.1` + a backend that does not signal `SETTINGS_ENABLE_CONNECT_PROTOCOL=1`, would succeed if HTTP/1.1 were used.

We suggest the following solutions:
* Envoy can have a behaviour change: remember if a h2 endpoint/upstream doesn't signal SETTINGS_ENABLE_CONNECT_PROTOCOL (or allow_connect is false) but ALPN includes `http1.1` can from that point on send websockets over http/1.1. This has the benefit of not introducing any new configuration key.
* if there is no provision for remembering the endpoint's history of signaling `SETTINGS_ENABLE_CONNECT_PROTOCOL` then a setting `always_use_h1_for_websocket: true` which forcefully uses http/1.1 for Websocket connections (if ALPN includes `http1.1` or ALPN is not sent), but permits HTTP/2 for all other requests.
* If we can remember whether SETTINGS_ENABLE_CONNECT_PROTOCOL was signaled, then an `auto_config` key `allow_websocket_h1_fallback: true` or `remember_upstream_connect_protocol_setting: true` which would permit Envoy to open a HTTP/1.1 connection for websocket proxying in the presence of a HTTP/2 backend that does not signal RFC 8441 `SETTINGS_ENABLE_CONNECT_PROTOCOL=1` support.

[optional *Relevant Links*:]
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-http2protocoloptions
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/upgrades.html
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-http2protocoloptions

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.