agent-substrate / agent-substrate/substrate

e2e lane flakes: worker-pool saturation and memory-pressure kills on the single-node CI cluster

Open
#1,133 2 comments 0 reactions 0 assignees View on GitHub
area/dev-infra area/tests kind/bug
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

The pr-workflow e2e job fails intermittently on main and on unrelated PRs. The failures look varied but reduce to two classes, both rooted in how much the lane runs concurrently on one ubuntu-latest kind node.

**Class 1**: worker-pool saturation (most frequent). ResumeActor fails with ResourceExhausted: no free workers available. Suite packages run concurrently (up to GOMAXPROCS), and the demo and metrics suites share the 3-worker counter demo pool, so timing overlap exhausts the pool and lifecycle tests fail immediately. Recent failed main runs contain 50+ occurrences each: 32516335516 (https://github.com/agent-substrate/substrate/actions/runs/32516335516), 32429542405 (https://github.com/agent-substrate/substrate/actions/runs/32429542405). The control plane already treats this error as wait-and-retry (the router's parking resumer does exactly that); #1132 makes the e2e suites do the same.

**Class 2**: sandbox processes killed under node pressure. Distinct symptoms, one shape — something SIGKILLs processes mid-test while the surrounding services stay healthy and report clean errors up the chain:
- runsc restore: signal: killed / runsc create: signal: killed during actor resume (identity suite; on main in 32436004611 (https://github.com/agent-substrate/substrate/actions/runs/32436004611) with elapsed times clustered at 30.4–30.5s across three separate failures, which may point at a watchdog rather than pure OOM timing — unconfirmed);
- a worker's ateom socket vanishing between actor start and suspend (32530161712 (https://github.com/agent-substrate/substrate/actions/runs/32530161712)).

Worker pods have no resource requests (BestEffort — first killed under pressure) and no liveness probes; the node also carries the full observability stack (jaeger, prometheus, otel collector), valkey, postgres, and every suite's fixture pools simultaneously.

**Ideas on how to reduce test flakiness**:
1. Retry saturation in the suites — #1132.
2. Drop the observability stack from the CI install (nothing in CI asserts on it) — likely several hundred MB back.
3. Add swap/zram on the runner — turns OOM kills into brief slowdowns.
4. Cap suite parallelism (go test -p 2) — helps both classes, costs wall clock.
5. Give the metrics suite its own pool instead of sharing the counter demo pool.
6. Resource requests on worker pools — converts mid-test kills into visible scheduling failures.
7. Shard the e2e job across two runners, or move to a larger runner.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.