Better signal for Envoy readiness in Kubernetes when using xDS
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 428
Description
*Title*: *Better signal for Envoy readiness in Kubernetes when using xDS*
*Description*:
When running Envoy as a sidecar proxy in Kubernetes, you often want to wait to mark the container as "ready" until Envoy has received a fully hydrated config via xDS. Usually, this means waiting for one of more dynamic resources to be present in the Admin interface's `/config_dump` response.
There does exist a `/ready` endpoint on Envoy's admin interface [1] however in my testing, it can still return OK even if the xDS configs aren't fully "hydrated" aka, all dynamic resources aren't yet present (which makes sense, as Envoy has no way to know what config _should_ be loaded without explicitly telling it).
What I'd like to propose is some way to give a hint to Envoy in the query params of the Admin request what configs to expect, and then return a response accordingly (either a 200 OK or some non-200 code) which can be used as a Kubernetes http readiness probe [3].
I had two initial thoughts, but open to other suggestions:
*Config Dump Endpoint Change:*
Add an optional query param to the `/config_dump` endpoint e.g. `/config_dump?expect=1` which will return some non-200 status (e.g. a 404) when `?name_regex={}` does not return the expect number of resources [2].
Today, the `/config_dump` endpoint _almost_ provides all the necessary params to perform the check I want. For example, if you want to wait for a specific dynamic listener to be present, you could do `/config_dump?resource=dynamic_listener&mask=name&name_regex=MY_LISTENER` however if the listener is not present, the endpoint still returns a `200 OK` status with an empty response, which would still pass a Kubernetes probe check. With `?expect={}`, we could trivially check on the server side the # of resources which matched the query, and return a non-200 status if they are not equal.
This is probably the most minimally invasive change, since most of the required query params for matching resources already exist on the config dump endpoint, but it also is adding functionality onto this endpoint for things it wasn't really intended for (readiness) so I can see not wanting to add this here, hence my next proposal:
*Ready Endpoint Change:*
Add the existing resource selector query params from the `/config_dump` endpoint as optional params to the `/ready` endpoint, and simply return a non-200 status when the specified query param resource matchers return no matches.
This would increase complexity of the ready endpoint quite a bit, but from a feature standpoint, it probably makes more sense to have this functionality on the `/ready` endpoint, since from my understanding this endpoint was intended for exactly this use case.
*Relevant Links*:
1. [ready endpoint](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#get--ready)
2. [config_dump endpoint](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#get--config_dump?name_regex=)
3. [kubernetes http probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request)
Contributor guide
Assessment
This issue has not been assessed yet.