hashicorp / hashicorp/consul

Mechanism to drain Envoy connections

Open
#8,304 1 comment 7 reactions 0 assignees View on GitHub
theme/connect theme/operator-usability
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

If users need to take a proxy/gateway out of commission there is no great way to do that at the moment.

The primary method for graceful shutdowns of Envoy is to use the [/healthcheck/fail](https://www.envoyproxy.io/docs/envoy/v1.11.2/operations/admin#operations-admin-interface-healthcheck-fail) endpoint (See [ENVOY-1990](https://github.com/envoyproxy/envoy/issues/1990)).

Calling this endpoint will lead Envoy to:
1. Send failed health check responses to downstream proxies
2. Start drain closing connections

However, we do not use Envoy's active health checking at the moment, so step 1 would do nothing. Connect proxies determine the health of upstream proxies through Consul, not through direct checks.

Consul should enable graceful connection draining somehow. Here are a few options:

1. Modify Consul checks on Envoy proxies to be HTTP checks rather than TCP. Listeners would have an [HTTP check filter](https://www.envoyproxy.io/docs/envoy/v1.11.2/intro/arch_overview/upstream/health_checking#arch-overview-health-checking-filter) in [no pass through mode](https://www.envoyproxy.io/docs/envoy/v1.11.2/intro/arch_overview/upstream/health_checking#http-health-checking-filter). Then, whenever a user calls `/healthcheck/fail`, a 503 would be returned whenever Consul probes the proxy. (Not sure if stacking this HTTP check filter on a TCP listener would work)

2. Add a new Consul check type that is only set to passing/failing via a manual toggle. This would be similar to a TTL check, except without an interval. This check type would be used for auto-registered Connect proxies and setting it to critical would prevent Connect traffic from being sent to that proxy. Users or Consul could then send a request to [/drain_listeners](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--drain_listeners) so Envoy sends `Connection: close` on HTTP request completions.

3. Add HTTP endpoints to `/agent/checks/` that enable users to force proxy TCP checks to become critical or passing. Making a check critical would disable the check runner, and making it passing would re-enable it. The listener draining side-effect from option 2 is also an option here.

Contributor guide

Open the contributing guide

Research direction

Start with the three proposed approaches and the referenced /healthcheck/fail, /drain_listeners, and /agent/checks/ endpoints. First determine which connection-draining mechanism Consul should support; done requires an agreed design and corresponding behavior for removing Envoy proxies from Connect traffic without dropping active connections.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.