envoyproxy / envoyproxy/envoy

xDS NACK loop between when switching from gRPC RDS to file-based RDS

Open
#10,046 7 comments 0 reactions 1 assignee Claimed by @wgallagher View on GitHub
area/xds help wanted investigate
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Title*: *NACK loop when switching from gRPC RDS to file-based RDS*

**Context**: we use java-control-plane, and non-incremental ADS.

**Note**: I'm not sure if this is a bug in Envoy, java-control-plane, or perhaps working as intended and we need to work around it in our custom control plane wrapping java-control-plane.

*Description*:

When transitioning the config source in [RDS](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/network/http_connection_manager/v2/http_connection_manager.proto#envoy-api-msg-config-filter-network-http-connection-manager-v2-rds) config from gRPC based to file path based, Envoy started endless logging lines like the following.

```
2020-02-12_22:59:02.92713 [2020-02-12 22:59:02.927][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
2020-02-12_22:59:02.94401 [2020-02-12 22:59:02.943][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
2020-02-12_22:59:02.96188 [2020-02-12 22:59:02.961][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
```

I believe what happened is this:

1) Envoy was operating using gRPC RDS
2) Our custom integration with java-control-plane saved a new configuration snapshot into the java-control-plane Cache. The snapshot included LDS config pointing the HCM to an RDS file. It _also_ included routes which was not necessary but for simplicity we always generate them.
3) java-control-plane responds with new Listener, Envoys switches to file-based RDS. (I'm unclear on what happens to the open RDS watches at this point).
4) java-control-plane responds with routes
5) Envoy hits [this code](https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_mux_impl.cc#L158), seemingly queuing another RDS request with an old version (NACK)
6) java-control-plane responds again
7) repeat 4-6

This loop did not stop until Envoy was restarted which breaks the ADS streams.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.