agent-substrate / agent-substrate/substrate
atenet: Envoy sizes its worker pool from the host CPU count
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 316
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 287
Description
## What happened
I ran an agent-substrate cluster on a 128 CPU node, and received these errors:
```
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
```
Both atenet pods crashlooped. The Go container next to each Envoy stayed healthy the whole time, which is why this reads like an atenet bug rather than a sizing problem.
Envoy picks its worker count from the host CPU count. Kubernetes CPU limits are CFS quota and don't constrain that, so on a big node Envoy starts a worker per host thread and burns through the default nofile before it finishes booting.
## What I did
I solved it by pinning the worker count instead of letting Envoy pick one, across all three Envoy containers:
- `--concurrency 4` on the Envoy container in `atenet-router.yaml` and `atenet-egress.yaml`. This is the change I ran on the box, and it cleared the crashloop.
- The same flag on `atenet-egress-with-sdsmint.yaml`. That file runs a third Envoy, selected by `--experimental-use-sdsmint` rather than by an overlay, so it inherited nothing from the other two. I only found it afterwards while checking coverage, and I have not run that path.
For the path I haven't run, I confirmed every Envoy container across the three manifests resolves to `--concurrency 4`, and that `base` and `kind` still build clean.
## Why this still needs work
The 4 is not measured. I picked something comfortably under the fd limit and a bit above what other projects use for sidecars. Nothing has been profiled against actor traffic.
What the number should track is the CPU the pod actually gets, and we can't express that today, because none of these containers declare CPU requests or limits.
## What to do instead
One option is to declare CPU requests on the three proxy containers, following what `kind/prometheus.yaml` already does: CPU requests, memory limits, no CPU limit.
Then pass the request in through the downward API:
```yaml
env:
- name: ENVOY_CONCURRENCY
valueFrom:
resourceFieldRef:
resource: requests.cpu
divisor: "1"
```
And read it. Kubernetes expands `$(VAR)` in `command` as well as `args`, so no wrapper script is involved:
```yaml
- --concurrency
- "$(ENVOY_CONCURRENCY)"
```
Another option is to be able to configure the concurrency itself using flags and not touching CPU at all.
## What this costs
- kind is the real constraint. `manifests/ate-install/kind/kustomization.yaml` pulls `../atenet-router.yaml` in directly with no resource patches, so whatever request we set lands on laptop clusters too. Either pick something small enough for kind and live with one thread everywhere, or patch it in the kind overlay and accept that the number differs per environment.
## What other projects do
- Istio derives it, reading `limits.cpu` into `ISTIO_CPU_LIMIT` through the downward API. It is only safe because their charts ship a default 2000m sidecar limit.
- Consul hardcodes 2, overridable per pod by annotation. Their values.yaml says to keep the number low for sidecars and high for edge proxies. Ours are edge.
- Envoy Gateway exposes `EnvoyProxy.spec.concurrency` and only emits `--concurrency` when you set it. No derivation at all.
## Related
- #665 measures atenet-router request capacity, which is what would give the worker count a real number instead of a guess.
Contributor guide
Assessment
This issue has not been assessed yet.