Mechanism to drain Envoy connections
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
If users need to take a proxy/gateway out of commission there is no great way to do that at the moment.
The primary method for graceful shutdowns of Envoy is to use the [/healthcheck/fail](https://www.envoyproxy.io/docs/envoy/v1.11.2/operations/admin#operations-admin-interface-healthcheck-fail) endpoint (See [ENVOY-1990](https://github.com/envoyproxy/envoy/issues/1990)).
Calling this endpoint will lead Envoy to:
1. Send failed health check responses to downstream proxies
2. Start drain closing connections
However, we do not use Envoy's active health checking at the moment, so step 1 would do nothing. Connect proxies determine the health of upstream proxies through Consul, not through direct checks.
Consul should enable graceful connection draining somehow. Here are a few options:
1. Modify Consul checks on Envoy proxies to be HTTP checks rather than TCP. Listeners would have an [HTTP check filter](https://www.envoyproxy.io/docs/envoy/v1.11.2/intro/arch_overview/upstream/health_checking#arch-overview-health-checking-filter) in [no pass through mode](https://www.envoyproxy.io/docs/envoy/v1.11.2/intro/arch_overview/upstream/health_checking#http-health-checking-filter). Then, whenever a user calls `/healthcheck/fail`, a 503 would be returned whenever Consul probes the proxy. (Not sure if stacking this HTTP check filter on a TCP listener would work)
2. Add a new Consul check type that is only set to passing/failing via a manual toggle. This would be similar to a TTL check, except without an interval. This check type would be used for auto-registered Connect proxies and setting it to critical would prevent Connect traffic from being sent to that proxy. Users or Consul could then send a request to [/drain_listeners](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--drain_listeners) so Envoy sends `Connection: close` on HTTP request completions.
3. Add HTTP endpoints to `/agent/checks/` that enable users to force proxy TCP checks to become critical or passing. Making a check critical would disable the check runner, and making it passing would re-enable it. The listener draining side-effect from option 2 is also an option here.
Contributor guide
Research direction
Start with the three proposed approaches and the referenced /healthcheck/fail, /drain_listeners, and /agent/checks/ endpoints. First determine which connection-draining mechanism Consul should support; done requires an agreed design and corresponding behavior for removing Envoy proxies from Connect traffic without dropping active connections.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100