kubernetes-sigs / kubernetes-sigs/devops-bench
feat(infra): Implement fail-closed model preflight, campaign launcher contract, and robust teardown
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Background & Problem Statement
Running evaluation campaigns across multiple models, harnesses, and clusters currently relies on manual shell scripts with several operational risks:
1. **Silent Model Fallback**: When judge or chaos model IDs are unset or misconfigured, the harness silently substitutes the agent's own model or an invalid ID, generating invalid evaluation scores. Preflight validation was often bypassed.
2. **Fragile Shell Launchers & Disk Exhaustion**: Launching multi-task matrices required dozens of bespoke shell scripts. Run workspaces (6–11 GB each) quickly filled host disks without automated cleanup, causing zero-byte result files and corrupted runs.
3. **Cluster Name Collisions**: Re-using shared cluster names allowed asynchronous teardown from stale runs to destroy active runs' fixtures.
4. **Chaos Injection Timeouts**: Scripted disruptions (e.g. load spikes on `optimize-scale`) were killed prematurely by aggressive command timeouts, scoring tasks without the disruption ever firing.
5. **OpenTofu Diagnostic Loss**: `tofu apply` failures swallowed `stderr`, preventing root-cause diagnosis in `results.json`.
### Proposed Architecture & Scope
1. **Fail-Closed Model Policy & Preflight CLI (`devops_bench/cli/`)**:
- `devops-bench preflight`: Validates judge and chaos model endpoints, Fortio presence, OpenClaw versions, available disk space, and IAM permissions before cluster provisioning.
- Refuse execution if judge model is unset or equals the model under test; remove preflight skip flags.
- Separate `JUDGE_API_KEY` and `CHAOS_API_KEY` configurations and enforce Vertex AI `global` location.
2. **Standardized Campaign Launcher Contract & CLI Surface**:
- Replace shell-based matrix runners with a structured environment variable contract implemented in Python.
- Introduce harness subcommands: `devops-bench launch `, `devops-bench status `, `devops-bench reap`.
- Store arm configurations as static preset files (`scripts/bastion/presets/.env`).
- Enforce run-unique cluster names carrying the run token and refuse relaunching live run IDs.
- Add automated workspace cleanup and PID-aware process management (`devops-bench reap`).
3. **Infrastructure Teardown & Chaos Hardening**:
- Harden chaos command execution: dynamically scale command timeouts (`declared + 60s`, capped at 900s), clamp load tool output, and introduce `chaos_invalidated` first-class status if disruption fails to fire.
- Forward `$HOME` to OpenTofu `local-exec` provisioners and preserve complete `stderr` output in `results.json` upon `tofu apply` failures.
- Add automated teardown script (`scripts/destroy-leaked-stack.sh`) with token matching to clean up orphaned clusters, service accounts, and load balancers.
### Acceptance Criteria
- [ ] Evaluation launch fails immediately if the judge model is unset or matches the agent under test.
- [ ] `devops-bench preflight` validates endpoint connectivity, binary dependencies, disk space, and permissions before provisioning.
- [ ] Matrix campaigns launch via `devops-bench launch ` with automated PID-aware reaping and status reporting.
- [ ] Cluster names are strictly unique per run; duplicate run ID launches are blocked.
- [ ] Chaos injections execute reliably or mark the run as `chaos_invalidated`.
- [ ] `tofu apply` failures preserve complete `stderr` diagnostics in `results.json`.
- [ ] `destroy-leaked-stack.sh` cleanly purges orphaned test clusters and cloud resources.
Contributor guide
Research direction
Start by reading the existing entry points under devops_bench/cli/ and the preset location scripts/bastion/presets/.env to map the launcher and preflight contract. Review scripts/destroy-leaked-stack.sh and the acceptance criteria; the work is done when the CLI, unique-run protection, cleanup, chaos status, and preserved OpenTofu diagnostics are implemented and verifiable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- cli, cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100