envoyproxy / envoyproxy/envoy

Do not close health check connections when draining

Open
#20,748 2 comments 0 reactions 0 assignees View on GitHub
area/health_checking help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Description*:
When Envoy is draining (via `/healthcheck/fail`) the [HttpConnectionManager will send a GOAWAY](https://github.com/envoyproxy/envoy/blob/main/source/common/http/conn_manager_impl.cc#L1434-L1445) or [Connection:close](https://github.com/envoyproxy/envoy/blob/main/source/common/http/conn_manager_impl.cc#L1476-L1485) on both application and health check connections – even if the health check requests are setting `Connection: keep-alive`.

In an active health checking setup, this causes unnecessary connection churn for health check connections – clients will continue to send health check requests, but will need to open a new connection each time.

This is problematic for servers that (a) receive a lot of health checks (b) use mTLS even for health check requests.

I would expect that if the [health check filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/health_check/v3/health_check.proto#envoy-v3-api-msg-extensions-filters-http-health-check-v3-healthcheck) is configured, Envoy would keep these connections alive.

*Repro steps*:
* Configure Envoy as below
* Drain Envoy with `healthcheck/fail`
* Send HTTP/1 health check request with `Connection: keep-alive`
* Observe 503 response with `Connection: close` header

*Config*:
* HttpConnectionManager
* HTTP Health Check filter

---

If this behaviour seems reasonable to folks, would love to hear how to approach adding this to Envoy.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.