LSD update draining is graceful, but not gradual
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
### Description
#### Short Description
When an LDS update occurs 100% of subsequent H1 responses immediately have `Connection: Close` added, and H2 Requests result in a GOAWAY. Connections without active requests are allowed to persist, and transactions are allowed to complete, but http codec messages to encourage a client to end their session are not introduced gradually, resulting in a large wave of reconnections.
#### Versions
All versions of Envoy
#### Long Description
We have noticed a difference between Listener draining behavior on shutdown/ `curl -X POST 'http://127.0.0.1:9901/drain_listeners?graceful&inboundonly` and LDS update with the following configuration:
```
--parent-shutdown-time-s 12000 \
--drain-time-s 10000 \
--drain-strategy gradual
```
| Event | Experience |
| ------------- | ------------- |
| shutdown or `curl -X POST 'http://127.0.0.1:9901/drain_listeners?graceful&inboundonly` | * Idle connections persist until `drain-time-s`
* New Connections are allowed
* Existing connections responses **gradually** have H1 `Connection: Close` and H2 GOAWAY added until 100% at `drain-time-s` |
| LDS update | * Idle connections persist until `drain-time-s`
* New Connections are allowed (on subsequent listener)
* 100% of subsequent H1 requests have `Connection: Close` added, and H2 Requests result in a GOAWAY **immediately** |
Looking at the code, it appears that a gradual drain manager only exists for server level draining: https://github.com/envoyproxy/envoy/blob/ad15deb3cf2bb8531deeb5bb8043426795b46ba6/source/server/drain_manager_impl.cc#L43-L86
While LDS update draining is graceful in the sense that it does not suddenly close all connections, it is not gradual: all subsequent requests for all sessions/clients have `Connection: Close` added, and H2 Requests result in a GOAWAY.
Chart of `listener_manager.lds.update_success` (green, dotted) vs `http.downstream_cx_drain_close` (red, solid)
Note how even though `drain-time-s 10000` suggests a ~3 hour drain time, all clients close their connections (envoy logs show `drain closing connection` on all requests, and all response headers contain `Connection: Close`).
### Why this is a problem
This can be a problem in scenarios with high numbers (>25k-100k-500k) of persistent connections per Envoy node with a high transaction rate. Clients close their connections on H1 `Connection: Close` / H2 GOAWAY on request, but immediately form a new connection in order to continue making requests. This results in a massive spike in legitimate connection attempts, which can overwhelm the box in terms of: rate limits (iptables, envoy), TLS handshakes, CPU, etc.
### What we think should be done
Envoy's LDS draining behavior on update should be either made consistent with server draining (`/drain_listeners?graceful&inboundonly`) i.e H1 `Connection: Close` and H2 GOAWAY should be introduced to responses gradually, or the listener should have a configurable `drain_strategy: gradual` which enables gradual probability of H1 `Connection: Close` and H2 GOAWAY per transaction.
Perhaps this could also be tackled at the same time as https://github.com/envoyproxy/envoy/issues/34500 . So that server and LDS draining are not only gradual, but can have differently configured lengths.
### Repro steps
1. Start envoy with the following options:
```
--parent-shutdown-time-s 12000 \
--drain-time-s 10000 \
--drain-strategy gradual
```
2. Start a simple LDS and serve a HTTP listener e.g. which can either serve a response directly (healthcheck filter, lua filter) or bounce off a cluster e.g. httpbin
```
resources:
- name: listener
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
common_http_protocol_options: {idle_timeout: 300.000s}
http2_protocol_options: {max_concurrent_streams: 100}
route_config:
name: route
virtual_hosts:
- name: route
domains:
- "*"
routes:
- match:
prefix: /
route:
cluster: httpbin
http_filters:
- name: Healthcheck178
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
headers:
- {name: x-envoy-provide-health, present_match: true}
pass_through_mode: false
- name: route
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
traffic_direction: INBOUND
"@type": type.googleapis.com/envoy.config.listener.v3.Listener
```
3. Start sending one or more persistent streams of H1 or H2 requests. I usually instantiate ~100-1000 clients.
```
import requests
import time
session = requests.Session()
while True:
response = session.get(
"https://127.0.0.1/anything",
verify=False,
headers={
"Host": "localreply.example.com",
},
)
if response.headers.get("Connection") == "close":
print("Envoy asked for connection to be closed")
time.sleep(1)
```
```
import requests
import time
import httpx
with httpx.Client(http2=True, verify=False) as client:
while True:
response = client.get(
"https://127.0.0.1/anything",
# verify=False,
headers={
"Host": "localreply.example.com",
}
)
time.sleep(1)
```
4. Mutate the listener e.g. change `name: Healthcheck178` to `name: Healthcheck179`. I did this by simply incrementing the name of the filter each LDS request.
5. Observe in Python or Wireshark that 100% of sessions/clients immediately H1 have `Connection: Close` added, and H2 Requests result in a GOAWAY on their next request, and the following is printed for all clients by the python script:
```
Envoy asked for connection to be closed
```
If you print the headers, all requests from all clients/sessions will immediately have:
Note: Each line represents a request sent by a different client session (Not a single client ignoring `Connection: Close`)
6. Run the experiment again without modifying the Listener, but this time call `curl -X POST 'http://127.0.0.1:9901/drain_listeners?graceful&inboundonly`
7. Notice that requests have a random chance of H1 `Connection: Close` / H2 GOAWAY on request, increasing to 100% likelihood at the end of `--drain-time-s 10000` i.e clients are able to send one to many requests before being asked to close the connection.
### Admin and Stats Output
`listener_manager.total_filter_chains_draining: 294` < increments every time LDS update occurs. As expected,
`downstream_cx_drain_close: XX` < immediately spikes to be the same number as the number of active requests.
### Logs
100% requests after LDS have this log line:
```
{"time": "2024-07-05 10:52:07.347", "level": "debug", "type": "http", "message": "[Tags: \"ConnectionId\":\"3116\",\"StreamId\":\"11248567122019182204\"] drain closing connection"}
```
Contributor guide
Assessment
This issue has not been assessed yet.