envoyproxy / envoyproxy/envoy

control plane fail-static backup: XDS fallback to low dependency config store when control plane is down

Open
#21,644 7 comments 0 reactions 0 assignees View on GitHub
area/configuration area/xds enhancement no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

*Title*: control plane fail-static backup: XDS fallback to low dependency config store when control plane is down

*Description*:
In outages like grpc lib error, or network issue, control plane global outages, we'd hope the proxy server to still work with existing config, but the config would go away if the running Envoy starts, in which case the dataplane is cascadingly down due to control plane outage.

This is extremely bad if a control plane outage is coupled with a data-plane update, that means we'd loose all our server capacity(zonal or even global).

while waiting for the control plane to come back, we'd like to have a fail-static config backup store that could allow restarted Envoys to load [possibly degraded/stale] config, serve the traffic as if control plane is just slow in delivering config data.

Ideally this "fail static" backup config store would:

* Automatically kick in on connection failures (retries drained) to management servers,
* Be reliable/high-available, most likely zonal-available( i.e. it's available from within the same zone).
Preferably "disk like" so that it could survive outages caused by say GRPC lib or network outage. (For cloud users, the high-available GCS server/Blob-store could be such a storage layer.
For Envoy Mobile, this could be the local cache layer before control plane connection is not available.
* Config data on the storage would be sharded, or mixed, we would need some mechanism to "filter " against the resources streams of the data.

@adisuissa @htuch

[optional *Relevant Links*:]
>Any extra documentation required to understand the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.