inotify watcher performance issues
- Lingua principale
- C++
- Stelle
- 28.9k
- Fork
- 5.6k
- Merge medio
- 1g 22h
- PR unite (30g)
- 430
Descrizione
*Title*: *inotify watcher is O(N) of watches for each file change*
*Description*:
When using the filesystem watcher for EDS configuration, I noticed significant performance overhead for configuration changes. With a large number of clusters (i've tested with ~1000) updating somewhat frequently (I've tested updating them all every 10s), Envoy ends up spending a significant portion of its time simply responding to inotify events. I think i've narrowed it town to a few factors at play.
* The inotify watcher watches the [directory](https://github.com/envoyproxy/envoy/blob/master/source/common/filesystem/inotify/watcher_impl.cc#L33) instead of the file itself. This means that what should be O(1) for a single file change, ends up being O(N) since any file change will trigger all watchers for that directory.
* In addition to the previous point, watching the directory causes even non-watched files to trigger inotify events. This was particularly bad in my case because we were writing temporary files into the same directory
* The inotify watcher watches `IN_ALL_EVENTS`, which includes irrelevant events like `IN_ACCESS`.
I've managed to work around this by having each EDS configuration in its own directory, not writing temp files in those directories, and minimizing accesses and other file operations on these files.
I have not tried this using the `kqueue` watcher.
*Repro steps*:
Create a large number of clusters with an eds_config file for each:
```
# clusters.yaml jinja template
resources:
{%- for service in services %}
- "@type": "type.googleapis.com/envoy.api.v2.Cluster"
name: {{ service }}.cluster
type: EDS
eds_cluster_config:
eds_config:
path: /tmp/xds/endpoints/{{ service }}.yaml
{%- endfor %}
```
You then just need a basic endpoint file for each cluster (you don't even need endpoints)
```
# {{ service }}.yaml
resources:
- "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
cluster_name: {{ service }}
endpoints:
- lb_endpoints:
```
I was recreating all of these files every 10 seconds or so. Clearly this is the wrong thing to do, but for the first version, it was sufficient.
I've uploaded a pprof callgraph from the cpuprofiler output [here](https://gist.github.com/bplotnick/39393343d92fb6e975e40da82d643e6b). It's in svg format, so you'll need to open the image (e.g. on Chrome, right click and hit "show image in new tab")
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.