envoyproxy / envoyproxy/envoy

Filesystem xDS races on updates leading to incorrect rejections

Open
#31,678 6 comments 0 reactions 0 assignees View on GitHub
area/xds bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
437

Description

>What issue is being seen? Describe what should be happening instead of the bug, for example: Envoy should not crash, the expected value isn't returned, etc.

I am observing occasional situations where my application unexpectedly fails to come online correctly, as requests to backends that are up, healthy, and registered with Envoy via the [filesystem xDS](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#filesystem-subscriptions) are rejected. Here's some simplified code from the startup program:

```
while True:
services_to_add = poll_for_new_services()
if services_to_add:
for service_name in services_to_add:
cluster = f"service_{service_name}"
# creates a config.cluster.v3.Cluster to be written to the CDS file
envoy_cfg.add_cluster(cluster, service_name)
# creates a config.route.v3.Route to be added to the main config.listener.v3.Listener in the LDS file
envoy_cfg.add_route(f"/{grpc_service}", cluster)
# refresh the CDS and LDS files
# note CDS is refreshed first because an LDS update will fail if it mentions any unknown clusters
envoy_cfg.refresh_cds()
envoy_cfg.refresh_lds()
time.sleep(POLLING_INTERVAL)
```

The note there might give it away, but I'm seeing fairly frequent `lds.update_rejected` events despite this ordering; i.e. Envoy will potentially process an LDS file update before an earlier CDS update. Generally this has proven benign because a subsequent loop manages to be read in the correct order and succeed, but occasionally the final time it discovers new services the writes race and Envoy ends up in a state where the LDS update is rejected, _then_ the CDS update is accepted, leaving the application broken even though the configs on-disk are compatible.

It's not shocking that these writes and reads could race, but it is very unfortunate that Envoy rejects LDS updates that are actually valid with no clear way to recover from this state. Ideally Envoy would fail more gracefully when given an incompatible route, or re-evaluate the LDS config it rejected when it observes a new CDS config.

I'm very much open to suggestions for other ways to implement this, but IMO it's a bug in Envoy that you can enter (and become stuck in) this state.

*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

Write new clusters and routes to a pair of CDS and LDS files repeatedly, CDS first, and observe rejected updates even though the file contents on-disk are never incompatible with one another.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.