aws / aws/aws-app-mesh-roadmap

Bug: Filtering of statistics leads to Envoy missing configuration

Open
#283 0 comments 0 reactions 1 assignee Claimed by @LancerRainier View on GitHub
Blocked on an Envoy fix Bug Docs
Dominant language
No language data
Stars
344
Forks
24
PR merge metrics
No merged PRs in 30d

Description

**Summary**

When customizing statistics emitted by Envoy via inclusion/exclusion filters, for example

```
stats_config:
stats_matcher:
inclusion_list:
patterns:
- safe_regex:
regex: "*rq*"
```

Envoy will enter a state where it will no longer properly synchronize with the App Mesh state of the world. This is due to an existing issue in Envoy where they leverage [statistics for control flow internally](https://github.com/envoyproxy/envoy/issues/9856). By filtering out certain statistics, Envoy is not able to determine when clusters are ready and process received configuration correctly.

**Steps to Reproduce**

Setting a customized stats configuration that includes stats filters.

**Are you currently working around this issue?**

Overall, our recommendation is to not perform any filtering of statistics in Envoy and instead leverage. Instead, we recommend reducing the overall volume of metrics emitted by leveraging the `ENVOY_STATS_FLUSH_INTERVAL` environment variable to reduce stats emission period (the default is 6 seconds).

If absolutely necessary, we've found that adding the following inclusion pattern

```
- suffix: warming_clusters
```

appears to resolve the problem, but we cannot guarantee there won't be future issues.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.