Local Rate limit definition is not working with Escape-Hatch Overrides configuration in Consul Envoy configuration
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Trying to registering sidecar into consul using hcl file and **consul services register** command with Local Rate limit definition inside filter level of Escape-Hatch Overrides configuration, throwing error following :
_[warning][config] [./source/common/config/grpc_stream.h:189] DeltaAggregatedResources gRPC config stream closed since 65898ms ago: 14, failed to generate all xDS resources from the snapshot: failed to generate xDS resources for "type.googleapis.com/envoy.config.listener.v3.Listener": can't unmarshal Any nested proto *envoy_config_listener_v3.Listener: can't unmarshal Any nested proto *envoy_extensions_filters_network_http_connection_manager_v3.HttpConnectionManager: unknown message type "envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit"_
**Definition:**
Try to implement rate-limiting [feature](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/local_rate_limit_filter#config-http-filters-local-rate-limit) of envoy which is running inside Consul infra. Since Consul have limited option to configure envoy we have chosen Escape-Hatch Overrides [configuration](https://www.consul.io/docs/connect/proxies/envoy#escape-hatch-overrides). We have defined all the config in hcl file and used that file to register the envoy sidecar service. Facing issue while updated with rate-limiting configuration inside filter level of Listener.
Tried three possible solution and none of the config is working, as following
1. Define LocalRateLimit filter inside http_filters of HttpConnectionManager [config-api-services.hcl](https://gist.github.com/subhajit-csc/d4e7b707e3b578c5547b4a90234e304a#file-config-api-services-hcl) . ### This case is applicable of inbound listener of sidecar
2. Define LocalRateLimit filter inside route level [config-api-services_with_route_level.hcl](https://gist.github.com/subhajit-csc/d4e7b707e3b578c5547b4a90234e304a#file-config-api-services_with_route_level-hcl). ### This case is applicable of inbound listener of sidecar
3. Tried to add same set of filter configuration in deferent field envoy_listener_json which is also another escape hatch configuration for outbound call [config-api-services_with_envoy_listener_json.hcl](https://gist.github.com/subhajit-csc/d4e7b707e3b578c5547b4a90234e304a#file-config-api-services_with_envoy_listener_json-hcl). ### In this case we want to limit all the call from sidecar to service via 8080 port.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Run consul services register -token {{ service_token }} {{ sidecar_config_path }}/api_service.hcl
1. Run sidecar docker compose or docker run
1. Check sidecar docker container log
### Operating system and Environment details
Consul Version : 1.12.2
Envoy Version: envoy:v1.20.0
### Log Fragments
_storesidecar_1 | [2022-06-08 10:07:09.535][7][info][main] [source/server/server.cc:393] response trailer map: 160 bytes: grpc-message,grpc-status
storesidecar_1 | [2022-06-08 10:07:09.565][7][info][main] [source/server/server.cc:740] runtime: {}
storesidecar_1 | [2022-06-08 10:07:09.566][7][info][admin] [source/server/admin/admin.cc:135] admin address: 0.0.0.0:19000
storesidecar_1 | [2022-06-08 10:07:09.567][7][info][config] [source/server/configuration_impl.cc:127] loading tracing configuration
storesidecar_1 | [2022-06-08 10:07:09.567][7][info][config] [source/server/configuration_impl.cc:87] loading 0 static secret(s)
storesidecar_1 | [2022-06-08 10:07:09.567][7][info][config] [source/server/configuration_impl.cc:93] loading 4 cluster(s)
storesidecar_1 | [2022-06-08 10:07:09.754][7][info][config] [source/server/configuration_impl.cc:97] loading 2 listener(s)
storesidecar_1 | [2022-06-08 10:07:09.762][7][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.filters.http.ext_authz.v3.ExtAuthz Using the default now-deprecated value AUTO for enum 'envoy.extensions.filters.http.ext_authz.v3.ExtAuthz.transport_api_version' from file ext_authz.proto. This enum value will be removed from Envoy soon so a non-default value must now be explicitly set. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
storesidecar_1 | [2022-06-08 10:07:09.762][7][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.filters.http.ext_authz.v3.ExtAuthz Using the default now-deprecated value AUTO for enum 'envoy.extensions.filters.http.ext_authz.v3.ExtAuthz.transport_api_version' from file ext_authz.proto. This enum value will be removed from Envoy soon so a non-default value must now be explicitly set. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
storesidecar_1 | [2022-06-08 10:07:09.780][7][info][config] [source/server/configuration_impl.cc:109] loading stats configuration
storesidecar_1 | [2022-06-08 10:07:09.781][7][info][main] [source/server/server.cc:836] starting main dispatch loop
storesidecar_1 | [2022-06-08 10:07:09.791][7][info][runtime] [source/common/runtime/runtime_impl.cc:449] RTDS has finished initialization
storesidecar_1 | [2022-06-08 10:07:09.791][7][info][upstream] [source/common/upstream/cluster_manager_impl.cc:202] cm init: initializing cds
storesidecar_1 | [2022-06-08 10:07:09.792][7][warning][main] [source/server/server.cc:715] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
storesidecar_1 | [2022-06-08 10:07:24.791][7][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster
storesidecar_1 | [2022-06-08 10:07:24.791][7][info][upstream] [source/common/upstream/cluster_manager_impl.cc:206] cm init: all clusters initialized
storesidecar_1 | [2022-06-08 10:07:24.791][7][info][main] [source/server/server.cc:817] all clusters initialized. initializing init manager
storesidecar_1 | [2022-06-08 10:07:39.792][7][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener
storesidecar_1 | [2022-06-08 10:07:39.792][7][info][config] [source/server/listener_manager_impl.cc:779] all dependencies initialized. starting workers
storesidecar_1 | [2022-06-08 10:07:50.605][7][warning][config] [./source/common/config/grpc_stream.h:189] DeltaAggregatedResources gRPC config stream closed since 40803ms ago: 14, failed to generate all xDS resources from the snapshot: failed to generate xDS resources for "type.googleapis.com/envoy.config.listener.v3.Listener": can't unmarshal Any nested proto *envoy_config_listener_v3.Listener: can't unmarshal Any nested proto *envoy_extensions_filters_network_http_connection_manager_v3.HttpConnectionManager: unknown message type "envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit"_
Contributor guide
Research direction
Reproduce the issue with `consul services register` and the sidecar using the three referenced configurations: config-api-services.hcl, config-api-services_with_route_level.hcl, and config-api-services_with_envoy_listener_json.hcl. Start with the Envoy logs showing the unknown LocalRateLimit message type; done means the intended rate-limit configuration is accepted without the xDS listener generation error.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100