envoyproxy / envoyproxy/envoy

inotify watcher performance issues

オープン
#3,308 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
area/perf help wanted
主要言語
C++
スター
28.9k
フォーク
5.6k
平均マージ
1日 22時間
マージ済み PR(30日)
430

説明

*Title*: *inotify watcher is O(N) of watches for each file change*

*Description*:
When using the filesystem watcher for EDS configuration, I noticed significant performance overhead for configuration changes. With a large number of clusters (i've tested with ~1000) updating somewhat frequently (I've tested updating them all every 10s), Envoy ends up spending a significant portion of its time simply responding to inotify events. I think i've narrowed it town to a few factors at play.

* The inotify watcher watches the [directory](https://github.com/envoyproxy/envoy/blob/master/source/common/filesystem/inotify/watcher_impl.cc#L33) instead of the file itself. This means that what should be O(1) for a single file change, ends up being O(N) since any file change will trigger all watchers for that directory.
* In addition to the previous point, watching the directory causes even non-watched files to trigger inotify events. This was particularly bad in my case because we were writing temporary files into the same directory
* The inotify watcher watches `IN_ALL_EVENTS`, which includes irrelevant events like `IN_ACCESS`.

I've managed to work around this by having each EDS configuration in its own directory, not writing temp files in those directories, and minimizing accesses and other file operations on these files.

I have not tried this using the `kqueue` watcher.

*Repro steps*:
Create a large number of clusters with an eds_config file for each:

```
# clusters.yaml jinja template
resources:
{%- for service in services %}
- "@type": "type.googleapis.com/envoy.api.v2.Cluster"
name: {{ service }}.cluster
type: EDS
eds_cluster_config:
eds_config:
path: /tmp/xds/endpoints/{{ service }}.yaml
{%- endfor %}
```
You then just need a basic endpoint file for each cluster (you don't even need endpoints)
```
# {{ service }}.yaml
resources:
- "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
cluster_name: {{ service }}
endpoints:
- lb_endpoints:
```

I was recreating all of these files every 10 seconds or so. Clearly this is the wrong thing to do, but for the first version, it was sufficient.

I've uploaded a pprof callgraph from the cpuprofiler output [here](https://gist.github.com/bplotnick/39393343d92fb6e975e40da82d643e6b). It's in svg format, so you'll need to open the image (e.g. on Chrome, right click and hit "show image in new tab")

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。