agent-substrate / agent-substrate/substrate
[P2] No fault-injection e2e tests for any critical failure scenario
- Lingua principale
- Go
- Stelle
- 1.8k
- Fork
- 316
- Merge medio
- 2g 43m
- PR unite (30g)
- 287
Descrizione
**Severity:** P2 (all P0/P1 issues invisible to CI; regressions will reach production)
**Component:** Test infrastructure — `internal/e2e/`
**Audit ID:** NET-10
---
## Summary
The e2e test suite covers happy-path lifecycle, identity after restore, network policy,
metrics emission, and request parking. There are no tests for any of the critical
failure scenarios identified in this audit: node deletion after pause (ISSUE-008),
ateapi crash mid-workflow (ISSUE-010), cold-actor first-request failure (ISSUE-009),
checkpoint re-invocation on torn-down workload (ISSUE-004), or Valkey data loss
(ISSUE-001). The longest test in the suite runs for ~10 seconds — the 30-second lock
TTL and 5-second ext_proc timeout boundary are never exercised. Every P0 and P1 issue
in this audit would have been caught immediately by a fault-injection test.
---
## What Exists
**File:** `internal/e2e/suites/` (confirmed directories):
- `demo/` — happy path lifecycle, actor routing, pod deletion (TestWorkerPodDeletion)
- `identity/` — identity cert after restore
- `networkpolicy/` — network isolation
- `metrics/` — OTel metric emission
- `parking/` — BudgetExhaustion, ParkThenServed sub-tests (~10s total)
**No suite covers:**
- Component crash + recovery (ateapi, atelet, atenet-router)
- Valkey restart or data loss
- Object store outage (GCS/S3 unavailable during checkpoint)
- PAUSED actor + node deletion
- Concurrent lifecycle RPCs on same actor
- Disk full on atelet node
---
## Impact
- P0/P1 regressions (ISSUE-001 through ISSUE-010) are undetectable in CI.
- A developer fixing one issue could re-introduce another with zero automated feedback.
- The stability plan (`SUBSTRATE-STABILITY-PLAN.md`) requires manual execution of all
experiments because no automated harness exists.
---
## Steps to Demonstrate the Gap
```bash
# Run the full e2e suite — observe zero fault-injection coverage
hack/run-e2e-kind.sh # should pass cleanly
grep -rn "kill\|crash\|fault\|inject\|SIGKILL\|SIGSTOP\|rollout restart" internal/e2e/
# Returns: zero results
```
---
## Expected Behavior
A fault-injection test suite in `internal/e2e/suites/stability/` covering:
| Test | Verifies |
|---|---|
| `TestAteSAPIKillMidResume` | Lock orphaned → actor unavailable ≤ TTL, then recovers |
| `TestWorkerPodDeletedWhileAtesapiDown` | Ghost worker cleaned at startup; actor recoverable |
| `TestObjectStoreOutageDuringSuspend` | Actor CRASHED, bucket objects cleaned up |
| `TestPausedActorNodeDisappears` | Actor transitions to CRASHED with clear error |
| `TestConcurrentRestoreSameActor` | Second Restore gets Aborted; first actor survives |
| `TestDiskFullOnAtelet` | Actor CRASHED with terminal error; not retried indefinitely |
| `TestValkeySinglePodRestart` | No data loss; actor survives pod restart |
---
## Suggested Fix
1. Add `stability/` test suite under `internal/e2e/suites/`.
2. Use `kubectl delete pod` + `kubectl wait` for component kill injection (no external
chaos framework needed — this matches the `stability/faults/` script approach in
`SUBSTRATE-STABILITY-PLAN.md`).
3. Add an invariant checker (per §2.1 of `SUBSTRATE-STABILITY-PLAN.md`) as a test
helper that runs after each fault scenario.
4. Gate these tests with a build tag `//go:build stability` so they run in nightly CI
but not in every presubmit.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.