No grpc stats present for mirrored traffic
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
*Title*: *Grpc stats not present for shadow/mirrored traffic*
*Description*:
>What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.
I don't see grpc stats present for mirrored traffic against my 'per instance' clusters (_see the Config section for more details_). I only see http level stats for these 'per instance' clusters.
*Config*:
>Include the config used to configure Envoy.
Envoy config
1. There is one 'main cluster' containing all endpoints for all our services
2. For each service instance/box we have a 'per instance cluster' that only contains the endpoints for that instance/box
Routing
1. We have one route for all incoming traffic that goes to our main cluster
Validation
My project validates newly spun up instances before their endpoints are added to this main cluster.
This is done by adding a mirror policy that mirrors incoming traffic on the main route to the per instance cluster for these newly spun up instances. Once the instance is validated its endpoints are added to the main cluster.
Listener config
We have grpc stats enabled using the HttpFilter
```
HttpFilter
.newBuilder()
.setName("envoy.filters.http.grpc_stats")
.setTypedConfig(Any.pack(FilterConfig.newBuilder().build()))
.build(),
```
*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.
Route mirror traffic to a newly spun up instance, observe no grpc stats for that per instance cluster when curling stats from envoy's stats api.
*Notes/Theories*:
I looked up the Envoy source code and I see that the implementation for [sending mirrored traffic](https://github.com/envoyproxy/envoy/blob/main/source/common/router/shadow_writer_impl.cc#L39) uses the [HttpAsyncClient](https://github.com/envoyproxy/envoy/blob/main/envoy/http/async_client.h#L68) only, the [GrpcAsyncClient](https://github.com/envoyproxy/envoy/blob/main/envoy/grpc/async_client.h) is not used currently, so this could be the cause of the metrics being missing.
My other theory is that mirrored traffic is not going through the http listener so it does not pass the the grpc_stats filter and collect those metrics.
Contributor guide
Assessment
This issue has not been assessed yet.