agent-substrate / agent-substrate/substrate

[P2] No fault-injection e2e tests for any critical failure scenario

Aberta
#614 0 comentários 0 reações 0 responsáveis Ver no GitHub
area/tests kind/feature prio/P2
Linguagem predominante
Go
Estrelas
1.8k
Forks
316
Merge médio
2d 43min
PRs com merge (30d)
287

Descrição

**Severity:** P2 (all P0/P1 issues invisible to CI; regressions will reach production)
**Component:** Test infrastructure — `internal/e2e/`
**Audit ID:** NET-10

---

## Summary

The e2e test suite covers happy-path lifecycle, identity after restore, network policy,
metrics emission, and request parking. There are no tests for any of the critical
failure scenarios identified in this audit: node deletion after pause (ISSUE-008),
ateapi crash mid-workflow (ISSUE-010), cold-actor first-request failure (ISSUE-009),
checkpoint re-invocation on torn-down workload (ISSUE-004), or Valkey data loss
(ISSUE-001). The longest test in the suite runs for ~10 seconds — the 30-second lock
TTL and 5-second ext_proc timeout boundary are never exercised. Every P0 and P1 issue
in this audit would have been caught immediately by a fault-injection test.

---

## What Exists

**File:** `internal/e2e/suites/` (confirmed directories):
- `demo/` — happy path lifecycle, actor routing, pod deletion (TestWorkerPodDeletion)
- `identity/` — identity cert after restore
- `networkpolicy/` — network isolation
- `metrics/` — OTel metric emission
- `parking/` — BudgetExhaustion, ParkThenServed sub-tests (~10s total)

**No suite covers:**
- Component crash + recovery (ateapi, atelet, atenet-router)
- Valkey restart or data loss
- Object store outage (GCS/S3 unavailable during checkpoint)
- PAUSED actor + node deletion
- Concurrent lifecycle RPCs on same actor
- Disk full on atelet node

---

## Impact

- P0/P1 regressions (ISSUE-001 through ISSUE-010) are undetectable in CI.
- A developer fixing one issue could re-introduce another with zero automated feedback.
- The stability plan (`SUBSTRATE-STABILITY-PLAN.md`) requires manual execution of all
experiments because no automated harness exists.

---

## Steps to Demonstrate the Gap

```bash
# Run the full e2e suite — observe zero fault-injection coverage
hack/run-e2e-kind.sh # should pass cleanly
grep -rn "kill\|crash\|fault\|inject\|SIGKILL\|SIGSTOP\|rollout restart" internal/e2e/
# Returns: zero results
```

---

## Expected Behavior

A fault-injection test suite in `internal/e2e/suites/stability/` covering:

| Test | Verifies |
|---|---|
| `TestAteSAPIKillMidResume` | Lock orphaned → actor unavailable ≤ TTL, then recovers |
| `TestWorkerPodDeletedWhileAtesapiDown` | Ghost worker cleaned at startup; actor recoverable |
| `TestObjectStoreOutageDuringSuspend` | Actor CRASHED, bucket objects cleaned up |
| `TestPausedActorNodeDisappears` | Actor transitions to CRASHED with clear error |
| `TestConcurrentRestoreSameActor` | Second Restore gets Aborted; first actor survives |
| `TestDiskFullOnAtelet` | Actor CRASHED with terminal error; not retried indefinitely |
| `TestValkeySinglePodRestart` | No data loss; actor survives pod restart |

---

## Suggested Fix

1. Add `stability/` test suite under `internal/e2e/suites/`.
2. Use `kubectl delete pod` + `kubectl wait` for component kill injection (no external
chaos framework needed — this matches the `stability/faults/` script approach in
`SUBSTRATE-STABILITY-PLAN.md`).
3. Add an invariant checker (per §2.1 of `SUBSTRATE-STABILITY-PLAN.md`) as a test
helper that runs after each fault scenario.
4. Gate these tests with a build tag `//go:build stability` so they run in nightly CI
but not in every presubmit.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.