kubernetes-sigs / kubernetes-sigs/devops-bench

feat(infra): Implement fail-closed model preflight, campaign launcher contract, and robust teardown

Open
#166 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Background & Problem Statement
Running evaluation campaigns across multiple models, harnesses, and clusters currently relies on manual shell scripts with several operational risks:
1. **Silent Model Fallback**: When judge or chaos model IDs are unset or misconfigured, the harness silently substitutes the agent's own model or an invalid ID, generating invalid evaluation scores. Preflight validation was often bypassed.
2. **Fragile Shell Launchers & Disk Exhaustion**: Launching multi-task matrices required dozens of bespoke shell scripts. Run workspaces (6–11 GB each) quickly filled host disks without automated cleanup, causing zero-byte result files and corrupted runs.
3. **Cluster Name Collisions**: Re-using shared cluster names allowed asynchronous teardown from stale runs to destroy active runs' fixtures.
4. **Chaos Injection Timeouts**: Scripted disruptions (e.g. load spikes on `optimize-scale`) were killed prematurely by aggressive command timeouts, scoring tasks without the disruption ever firing.
5. **OpenTofu Diagnostic Loss**: `tofu apply` failures swallowed `stderr`, preventing root-cause diagnosis in `results.json`.

### Proposed Architecture & Scope
1. **Fail-Closed Model Policy & Preflight CLI (`devops_bench/cli/`)**:
- `devops-bench preflight`: Validates judge and chaos model endpoints, Fortio presence, OpenClaw versions, available disk space, and IAM permissions before cluster provisioning.
- Refuse execution if judge model is unset or equals the model under test; remove preflight skip flags.
- Separate `JUDGE_API_KEY` and `CHAOS_API_KEY` configurations and enforce Vertex AI `global` location.
2. **Standardized Campaign Launcher Contract & CLI Surface**:
- Replace shell-based matrix runners with a structured environment variable contract implemented in Python.
- Introduce harness subcommands: `devops-bench launch `, `devops-bench status `, `devops-bench reap`.
- Store arm configurations as static preset files (`scripts/bastion/presets/.env`).
- Enforce run-unique cluster names carrying the run token and refuse relaunching live run IDs.
- Add automated workspace cleanup and PID-aware process management (`devops-bench reap`).
3. **Infrastructure Teardown & Chaos Hardening**:
- Harden chaos command execution: dynamically scale command timeouts (`declared + 60s`, capped at 900s), clamp load tool output, and introduce `chaos_invalidated` first-class status if disruption fails to fire.
- Forward `$HOME` to OpenTofu `local-exec` provisioners and preserve complete `stderr` output in `results.json` upon `tofu apply` failures.
- Add automated teardown script (`scripts/destroy-leaked-stack.sh`) with token matching to clean up orphaned clusters, service accounts, and load balancers.

### Acceptance Criteria
- [ ] Evaluation launch fails immediately if the judge model is unset or matches the agent under test.
- [ ] `devops-bench preflight` validates endpoint connectivity, binary dependencies, disk space, and permissions before provisioning.
- [ ] Matrix campaigns launch via `devops-bench launch ` with automated PID-aware reaping and status reporting.
- [ ] Cluster names are strictly unique per run; duplicate run ID launches are blocked.
- [ ] Chaos injections execute reliably or mark the run as `chaos_invalidated`.
- [ ] `tofu apply` failures preserve complete `stderr` diagnostics in `results.json`.
- [ ] `destroy-leaked-stack.sh` cleanly purges orphaned test clusters and cloud resources.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing entry points under devops_bench/cli/ and the preset location scripts/bastion/presets/.env to map the launcher and preflight contract. Review scripts/destroy-leaked-stack.sh and the acceptance criteria; the work is done when the CLI, unique-run protection, cleanup, chaos status, and preserved OpenTofu diagnostics are implemented and verifiable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
cli, cloud, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.