agent-substrate / agent-substrate/substrate

atenet: Envoy sizes its worker pool from the host CPU count

Đang mở
#1,193 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/network kind/bug
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

## What happened

I ran an agent-substrate cluster on a 128 CPU node, and received these errors:

```
[warn] evutil_make_internal_pipe_: pipe: Too many open files
[err] evsig_init_: socketpair: Too many open files
```

Both atenet pods crashlooped. The Go container next to each Envoy stayed healthy the whole time, which is why this reads like an atenet bug rather than a sizing problem.

Envoy picks its worker count from the host CPU count. Kubernetes CPU limits are CFS quota and don't constrain that, so on a big node Envoy starts a worker per host thread and burns through the default nofile before it finishes booting.

## What I did

I solved it by pinning the worker count instead of letting Envoy pick one, across all three Envoy containers:

- `--concurrency 4` on the Envoy container in `atenet-router.yaml` and `atenet-egress.yaml`. This is the change I ran on the box, and it cleared the crashloop.
- The same flag on `atenet-egress-with-sdsmint.yaml`. That file runs a third Envoy, selected by `--experimental-use-sdsmint` rather than by an overlay, so it inherited nothing from the other two. I only found it afterwards while checking coverage, and I have not run that path.

For the path I haven't run, I confirmed every Envoy container across the three manifests resolves to `--concurrency 4`, and that `base` and `kind` still build clean.

## Why this still needs work

The 4 is not measured. I picked something comfortably under the fd limit and a bit above what other projects use for sidecars. Nothing has been profiled against actor traffic.

What the number should track is the CPU the pod actually gets, and we can't express that today, because none of these containers declare CPU requests or limits.

## What to do instead

One option is to declare CPU requests on the three proxy containers, following what `kind/prometheus.yaml` already does: CPU requests, memory limits, no CPU limit.

Then pass the request in through the downward API:

```yaml
env:
- name: ENVOY_CONCURRENCY
valueFrom:
resourceFieldRef:
resource: requests.cpu
divisor: "1"
```

And read it. Kubernetes expands `$(VAR)` in `command` as well as `args`, so no wrapper script is involved:

```yaml
- --concurrency
- "$(ENVOY_CONCURRENCY)"
```

Another option is to be able to configure the concurrency itself using flags and not touching CPU at all.

## What this costs

- kind is the real constraint. `manifests/ate-install/kind/kustomization.yaml` pulls `../atenet-router.yaml` in directly with no resource patches, so whatever request we set lands on laptop clusters too. Either pick something small enough for kind and live with one thread everywhere, or patch it in the kind overlay and accept that the number differs per environment.

## What other projects do

- Istio derives it, reading `limits.cpu` into `ISTIO_CPU_LIMIT` through the downward API. It is only safe because their charts ship a default 2000m sidecar limit.
- Consul hardcodes 2, overridable per pod by annotation. Their values.yaml says to keep the number low for sidecars and high for edge proxies. Ours are edge.
- Envoy Gateway exposes `EnvoyProxy.spec.concurrency` and only emits `--concurrency` when you set it. No derivation at all.

## Related

- #665 measures atenet-router request capacity, which is what would give the worker count a real number instead of a guess.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.