agent-substrate / agent-substrate/substrate
e2e lane flakes: worker-pool saturation and memory-pressure kills on the single-node CI cluster
- 主要語言
- Go
- 星號
- 1.8k
- 分支
- 316
- 平均合併
- 2 天 43 分鐘
- 30 天內合併 PR
- 287
描述
The pr-workflow e2e job fails intermittently on main and on unrelated PRs. The failures look varied but reduce to two classes, both rooted in how much the lane runs concurrently on one ubuntu-latest kind node.
**Class 1**: worker-pool saturation (most frequent). ResumeActor fails with ResourceExhausted: no free workers available. Suite packages run concurrently (up to GOMAXPROCS), and the demo and metrics suites share the 3-worker counter demo pool, so timing overlap exhausts the pool and lifecycle tests fail immediately. Recent failed main runs contain 50+ occurrences each: 32516335516 (https://github.com/agent-substrate/substrate/actions/runs/32516335516), 32429542405 (https://github.com/agent-substrate/substrate/actions/runs/32429542405). The control plane already treats this error as wait-and-retry (the router's parking resumer does exactly that); #1132 makes the e2e suites do the same.
**Class 2**: sandbox processes killed under node pressure. Distinct symptoms, one shape — something SIGKILLs processes mid-test while the surrounding services stay healthy and report clean errors up the chain:
- runsc restore: signal: killed / runsc create: signal: killed during actor resume (identity suite; on main in 32436004611 (https://github.com/agent-substrate/substrate/actions/runs/32436004611) with elapsed times clustered at 30.4–30.5s across three separate failures, which may point at a watchdog rather than pure OOM timing — unconfirmed);
- a worker's ateom socket vanishing between actor start and suspend (32530161712 (https://github.com/agent-substrate/substrate/actions/runs/32530161712)).
Worker pods have no resource requests (BestEffort — first killed under pressure) and no liveness probes; the node also carries the full observability stack (jaeger, prometheus, otel collector), valkey, postgres, and every suite's fixture pools simultaneously.
**Ideas on how to reduce test flakiness**:
1. Retry saturation in the suites — #1132.
2. Drop the observability stack from the CI install (nothing in CI asserts on it) — likely several hundred MB back.
3. Add swap/zram on the runner — turns OOM kills into brief slowdowns.
4. Cap suite parallelism (go test -p 2) — helps both classes, costs wall clock.
5. Give the metrics suite its own pool instead of sharing the counter demo pool.
6. Resource requests on worker pools — converts mid-test kills into visible scheduling failures.
7. Shard the e2e job across two runners, or move to a larger runner.
貢獻指南
評估
這個 Issue 還沒有評估資料。