agent-substrate / agent-substrate/substrate

[P2] No fault-injection e2e tests for any critical failure scenario

Đang mở
#614 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/tests kind/feature prio/P2
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

**Severity:** P2 (all P0/P1 issues invisible to CI; regressions will reach production)
**Component:** Test infrastructure — `internal/e2e/`
**Audit ID:** NET-10

---

## Summary

The e2e test suite covers happy-path lifecycle, identity after restore, network policy,
metrics emission, and request parking. There are no tests for any of the critical
failure scenarios identified in this audit: node deletion after pause (ISSUE-008),
ateapi crash mid-workflow (ISSUE-010), cold-actor first-request failure (ISSUE-009),
checkpoint re-invocation on torn-down workload (ISSUE-004), or Valkey data loss
(ISSUE-001). The longest test in the suite runs for ~10 seconds — the 30-second lock
TTL and 5-second ext_proc timeout boundary are never exercised. Every P0 and P1 issue
in this audit would have been caught immediately by a fault-injection test.

---

## What Exists

**File:** `internal/e2e/suites/` (confirmed directories):
- `demo/` — happy path lifecycle, actor routing, pod deletion (TestWorkerPodDeletion)
- `identity/` — identity cert after restore
- `networkpolicy/` — network isolation
- `metrics/` — OTel metric emission
- `parking/` — BudgetExhaustion, ParkThenServed sub-tests (~10s total)

**No suite covers:**
- Component crash + recovery (ateapi, atelet, atenet-router)
- Valkey restart or data loss
- Object store outage (GCS/S3 unavailable during checkpoint)
- PAUSED actor + node deletion
- Concurrent lifecycle RPCs on same actor
- Disk full on atelet node

---

## Impact

- P0/P1 regressions (ISSUE-001 through ISSUE-010) are undetectable in CI.
- A developer fixing one issue could re-introduce another with zero automated feedback.
- The stability plan (`SUBSTRATE-STABILITY-PLAN.md`) requires manual execution of all
experiments because no automated harness exists.

---

## Steps to Demonstrate the Gap

```bash
# Run the full e2e suite — observe zero fault-injection coverage
hack/run-e2e-kind.sh # should pass cleanly
grep -rn "kill\|crash\|fault\|inject\|SIGKILL\|SIGSTOP\|rollout restart" internal/e2e/
# Returns: zero results
```

---

## Expected Behavior

A fault-injection test suite in `internal/e2e/suites/stability/` covering:

| Test | Verifies |
|---|---|
| `TestAteSAPIKillMidResume` | Lock orphaned → actor unavailable ≤ TTL, then recovers |
| `TestWorkerPodDeletedWhileAtesapiDown` | Ghost worker cleaned at startup; actor recoverable |
| `TestObjectStoreOutageDuringSuspend` | Actor CRASHED, bucket objects cleaned up |
| `TestPausedActorNodeDisappears` | Actor transitions to CRASHED with clear error |
| `TestConcurrentRestoreSameActor` | Second Restore gets Aborted; first actor survives |
| `TestDiskFullOnAtelet` | Actor CRASHED with terminal error; not retried indefinitely |
| `TestValkeySinglePodRestart` | No data loss; actor survives pod restart |

---

## Suggested Fix

1. Add `stability/` test suite under `internal/e2e/suites/`.
2. Use `kubectl delete pod` + `kubectl wait` for component kill injection (no external
chaos framework needed — this matches the `stability/faults/` script approach in
`SUBSTRATE-STABILITY-PLAN.md`).
3. Add an invariant checker (per §2.1 of `SUBSTRATE-STABILITY-PLAN.md`) as a test
helper that runs after each fault scenario.
4. Gate these tests with a build tag `//go:build stability` so they run in nightly CI
but not in every presubmit.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.