envoyproxy / envoyproxy/gateway
gateway controller overloads envoy with many xDS updates
- 主要言語
- Go
- スター
- 3k
- フォーク
- 864
- 平均マージ
- 2日 2時間
- マージ済み PR(30日)
- 140
説明
*Description*:
>What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.
After upgrading to a newer version of the controller (v0.5.0 -> v1.2.4). We experienced an incident where a large number of pods being rolled concurrently over the course of a few hours caused envoy to get OOMd frequently (about every 10 minutes). We believe the primary cause of this was the gateway controller pushing many updates to envoy.
I think this was due to the addition of endpoint routing. Each update to an endpoint slice triggers an event which will cause a reconcile to run. We aren't using endpoint routing so any updates to endpoints should really be a noop on the config.
The controller should skip pushing updates in this case. Either by skipping events triggered by endpoint slices or by detecting that the envoy configuration has no changes.
I am curious to hear other ideas to address this issue as well. I am worried that there could be other cases where the controller could overload envoy with configuration updates. Maybe some sort of rate limiting or better detection of updates which cause no configuration change to envoy as I mentioned above would be a more robust solution.
*Repro steps*:
> Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.
- Configure an EnvoyProxy resource with `.spec.routingType: "Service"`
- Setup a route to a service with endpoints
- Roll the pods in the endpoint slice to create new endpoints
- Observe configuration updates pushed to envoy via controller logs or envoys lds metrics
*Environment*:
>Include the environment like gateway version, envoy version and so on.
gateway: v1.2.4
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。