envoyproxy / envoyproxy/envoy

Starting envoy with one file for CDS clusters, and one for EDS endpoints, with 400 clusters gets stuck handling inotify events.

Open
#5,807 3 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 20h
Merged PRs (30d)
428

Description

*Title*: Starting envoy with one file for CDS clusters, and one for EDS endpoints, with 400 clusters gets stuck handling inotify events.

*Description*:
Create a file for CDS containing 400 clusters in the format
```
- "@type": type.googleapis.com/envoy.api.v2.Cluster
name: my-service
connect_timeout: 30s
type: EDS
eds_cluster_config:
eds_config:
path: "/mnt/envoy/envoy/apiproxy/endpoints.yaml"
```
Then create one file (endpoints.yaml) which contains endpoints for all the clusters with the format:
```
- "@type": type.googleapis.com/envoy.api.v2.ClusterLoadAssignment
cluster_name: my-service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 10.3.33.2
port_value: 5349
- endpoint:
address:
socket_address:
address: 10.3.33.3
port_value: 5349
```

In the envoy config just use:
```
dynamic_resources:
cds_config:
path: "/mnt/envoy/envoy/apiproxy/clusters.yaml"
```

In the clean envoy docker ubuntu image latest, it will crash with the error
```
2019-01-31 07:29:48.247][001502][critical][assert] [source/common/filesystem/inotify/watcher_impl.cc:27] assert failure: inotify_fd_ >= 0.
[2019-01-31 07:29:48.247][001502][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Aborted, suspect faulting address 0x5de
[2019-01-31 07:29:48.247][001502][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<0> obj (If unsymbolized, use tools
/stack_decode.py):
```

If u raise
```
fs.inotify.max_queued_events
```
to a big value then it will keep on receiving inotify events, but won't start in any reasonable time (I waited a few minutes). It seems like each subsequent cluster starting its watch triggers inotify events for all previous Clusters. (which would make it O(n^2).

Creating different directories for each clusters endpoints (you can actually just copy the endpoints.yaml file containing endpoints for 400 services), so that each cluster watches its own directory makes envoy startup in 10-30s.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.