xDS NACK loop between when switching from gRPC RDS to file-based RDS
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
*Title*: *NACK loop when switching from gRPC RDS to file-based RDS*
**Context**: we use java-control-plane, and non-incremental ADS.
**Note**: I'm not sure if this is a bug in Envoy, java-control-plane, or perhaps working as intended and we need to work around it in our custom control plane wrapping java-control-plane.
*Description*:
When transitioning the config source in [RDS](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/network/http_connection_manager/v2/http_connection_manager.proto#envoy-api-msg-config-filter-network-http-connection-manager-v2-rds) config from gRPC based to file path based, Envoy started endless logging lines like the following.
```
2020-02-12_22:59:02.92713 [2020-02-12 22:59:02.927][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
2020-02-12_22:59:02.94401 [2020-02-12 22:59:02.943][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
2020-02-12_22:59:02.96188 [2020-02-12 22:59:02.961][212341][warning][config] [external/envoy/source/common/config/grpc_mux_impl.cc:153] Ignoring unwatched type URL type.googleapis.com/envoy.api.v2.RouteConfiguration
```
I believe what happened is this:
1) Envoy was operating using gRPC RDS
2) Our custom integration with java-control-plane saved a new configuration snapshot into the java-control-plane Cache. The snapshot included LDS config pointing the HCM to an RDS file. It _also_ included routes which was not necessary but for simplicity we always generate them.
3) java-control-plane responds with new Listener, Envoys switches to file-based RDS. (I'm unclear on what happens to the open RDS watches at this point).
4) java-control-plane responds with routes
5) Envoy hits [this code](https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_mux_impl.cc#L158), seemingly queuing another RDS request with an old version (NACK)
6) java-control-plane responds again
7) repeat 4-6
This loop did not stop until Envoy was restarted which breaks the ADS streams.
Contributor guide
Assessment
This issue has not been assessed yet.