Can't use internal redirect with websocket connection upgrade requests
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
Under certain circumstances I would like to redirect a websocket connection upgrade request from one private location to another, in which the initial connection upgrade request is internally redirected after receiving a 302 from the service host fielding the initial request. It appears Envoy should support this, but does not due to a specific condition in the internal redirect handling code.
The flow below is what I would like to have working:
```
+--------+ ---upgrade: websocket---> +-------+ ---upgrade: websocket---> +-------+
| client | | envoy | | srv_A |
+--------+ <------101 Upgraded----- +-------+ <-----302, Loc: srv_B---- +-------+
^ |
| |
| upgrade: websocket
| |
101 |
| v
+-------+
| srv_B |
+-------+
```
It looks from the Envoy docs like this should be possible using `internal_redirect_policy` and `upgrade_configs` for the route configs. To test this, I modified the docker-compose.yml from the [envoy websocket example](https://github.com/envoyproxy/envoy/tree/main/examples/websocket). I added a cluster containing a simple node server that just redirects to the websocat cluster (see config below), and added the internal redirect policy and upgrade config. However, Envoy fails to perform the internal redirect, and the internal 302 is passed to the downstream client. I have separately verified that I can do internal redirects when the `upgrade_configs` block is not included in the config.
After adding some additional trace messages to the code, it appears the `Filter::setupRedirect` method will not perform the internal redirect if the downstream request is not complete (`downstream_end_stream_ == false`). The downstream request will of course not be complete because this is an upgrade request.
I don't see a way around this using currently available configuration options. It looks to me like my use case could be supported with a slightly more refined conditional in `Filter::setupRedirect` based on the presence of upgrade_configs (or a new config flag enabling redirect support in combination with upgrades), but I'm not sufficiently experienced with the code at this point to be sure.
*Config*:
```yaml
# envoy-ws.yaml
admin:
address:
socket_address:
address: 0.0.0.0
port_value: 9001
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_ws_to_ws
route_config:
name: local_route
virtual_hosts:
- name: app
domains:
- "*"
routes:
- match:
prefix: "/websocat"
route:
cluster: service_ws
internal_redirect_policy:
max_internal_redirects: 5
upgrade_configs:
- upgrade_type: websocket
- match:
prefix: "/"
route:
cluster: redir
internal_redirect_policy:
max_internal_redirects: 5
upgrade_configs:
- upgrade_type: websocket
http_filters:
- name: envoy.filters.http.router
clusters:
- name: redir
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: redir
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: redir
port_value: 80
- name: service_ws
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_ws
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service-ws
port_value: 80
```
```yaml
# docker-compose.yaml
version: "3.7"
services:
proxy-ws:
image: envoy:parrot
ports:
- "9001:9001"
- "10000:10000"
volumes:
- ./envoy-ws.yaml:/etc/envoy.yaml
redir:
build:
context: .
dockerfile: Dockerfile.redir-server
hostname: redir
environment:
PORT: 80
TARGET_HOST: service-ws
service-ws:
image: solsson/websocat
hostname: service-ws
command: -E ws-listen:0.0.0.0:80 literalreply:'[ws] HELO'
```
*Logs*:
Full trace logs in this gist: [envoy_ws_upgrade_internal_redirect_failure_logs](https://gist.github.com/amortensen/d27e37a0b43cd804a8e79cdbfd4edf17#file-envoy_ws_upgrade_internal_redirect_failure_logs). Key excerpts:
```proxy-ws_1 | [2022-01-14 04:56:20.316][22][debug][conn_handler] [source/server/active_tcp_listener.cc:140] [C2] new connection from 172.18.0.1:62668
proxy-ws_1 | [2022-01-14 04:56:20.317][22][debug][http] [source/common/http/conn_manager_impl.cc:283] [C2] new stream
proxy-ws_1 | [2022-01-14 04:56:20.318][22][debug][http] [source/common/http/conn_manager_impl.cc:873] [C2][S17685677658886189605] request headers complete (end_stream=false):
proxy-ws_1 | ':authority', 'localhost:10000'
proxy-ws_1 | ':path', '/'
proxy-ws_1 | ':method', 'GET'
proxy-ws_1 | 'upgrade', 'websocket'
proxy-ws_1 | 'connection', 'Upgrade'
proxy-ws_1 | 'sec-websocket-key', '1cFsmiWI3V09wQQ8g+wgRQ=='
proxy-ws_1 | 'origin', 'http://localhost/'
proxy-ws_1 | 'sec-websocket-version', '13'
proxy-ws_1 |
proxy-ws_1 | [2022-01-14 04:56:20.318][22][debug][router] [source/common/router/router.cc:486] [C2][S17685677658886189605] cluster 'redir' match for URL '/'
...
proxy-ws_1 | [2022-01-14 04:56:20.320][22][trace][http] [source/common/http/http1/codec_impl.cc:869] [C2] message complete
proxy-ws_1 | [2022-01-14 04:56:20.320][22][trace][http] [source/common/http/http1/codec_impl.cc:877] [C2] Pausing parser due to upgrade.
proxy-ws_1 | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:655] [C2] parsed 181 bytes
proxy-ws_1 | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:582] [C2] direct-dispatched 0 bytes
proxy-ws_1 | [2022-01-14 04:56:20.321][22][trace][http] [source/common/http/http1/codec_impl.cc:1175] [C2] body size=0
...
proxy-ws_1 | [2022-01-14 04:56:20.325][22][trace][http] [source/common/http/http1/codec_impl.cc:1321] [C3] status_code 302
proxy-ws_1 | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1322] [C2][S17685677658886189605] upstream headers complete: end_stream=false
proxy-ws_1 | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1564] [C2][S17685677658886189605] attempting internal redirect
proxy-ws_1 | [2022-01-14 04:56:20.325][22][debug][router] [source/common/router/router.cc:1589] [C2][S17685677658886189605] Internal redirect failed
...
proxy-ws_1 | [2022-01-14 04:56:20.325][22][debug][http] [source/common/http/conn_manager_impl.cc:1472] [C2][S17685677658886189605] encoding headers via codec (end_stream=false):
proxy-ws_1 | ':status', '302'
proxy-ws_1 | 'location', 'http://service-ws/websocat'
proxy-ws_1 | 'date', 'Fri, 14 Jan 2022 04:56:20 GMT'
proxy-ws_1 | 'server', 'envoy'
proxy-ws_1 | 'connection', 'close'
proxy-ws_1 |
```
Contributor guide
Assessment
This issue has not been assessed yet.