kubernetes-sigs / kubernetes-sigs/devops-bench

feat(verification): Enhance deterministic verification engine with hold-window monitoring, fair budget sharing, and expanded verifier suite

Open
#168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Background & Problem Statement
Evaluating agents deterministically against real cluster state requires robust verification mechanisms and fair time budgeting:
1. **Flaky Convergence & Verification Budget Starvation**: In asynchronous Kubernetes workflows, dividing verification time evenly among waiting checks while counting instant `hold` checks in the divisor starved converge checks (e.g. giving 85.7s of a declared 120s window, causing incomplete coverage).
2. **Missing Post-Run Observation Windows (`hold` mode)**: The harness lacked a structured mechanism for post-run "hold" windows (asserting that a fixed state persists steadily over 180s/300s) and invariant monitoring (asserting that safety conditions were never breached during execution).
3. **Deadline Handoff Loss**: When parallel verification children reached a verdict near or at the shared deadline, parent processes discarded the verdict and reported an unobserved timeout.
4. **Missing or Incomplete Verifiers**: Tasks lacked standard verifiers for HTTP edge routing, GitOps repository synchronizations, in-pod execution output, and RBAC authorization.
5. **Masked Verification Errors**: Syntax errors in task YAMLs or underlying API exceptions were frequently masked as agent failures rather than surfacing as harness errors.

### Proposed Architecture & Scope
1. **Hold Mode & Safeguard Windows (`devops_bench/verification/hold.py`)**:
- Implement `mode: hold` for both objective verification (requiring a state to remain true across a sustained observation window) and safeguard verification (monitoring invariant stability).
- Implement structured fold logic (`_fold_sample`) to distinguish transient check retries from sustained trailing errors.
2. **Fair Budget Allocation & 1800s Standard (`devops_bench/verification/runner.py`)**:
- Adopt 1800s campaign verification standard.
- Allocate the remaining budget pool only across active converging entries, excluding zero-cost hold checks from the divisor.
- Preserve parallel child verdicts that arrive at or near the deadline.
3. **Comprehensive Verifier Suite (`devops_bench/verification/verifiers/`)**:
- `http_probe`: HTTP/HTTPS connectivity assertions with host headers, curl markers, status code validation, and bounded body matching.
- `git_repo_sync`: Assert Git repository commit state, branches, and tracked contents (with `across_matches` quantifier).
- `pod_exec`: Execute commands within target pods to verify runtime filesystem state and process outputs.
- `identity_preserved`: Assert resource ownership, metadata generation, and object immutability.
- `subject_access_review`: Verify Kubernetes RBAC authorization policies deterministically.
- `cloud_resource_property`: Verify managed cloud infrastructure attributes.
- `workspace_file`: Verify presence and contents of generated artifacts in the agent's workspace.
4. **Verification Result Integrity**:
- Fail loudly on unparseable YAML spec entries.
- Withhold correctness on verifier errors to keep infrastructure flakiness distinct from agent mistakes.

### Acceptance Criteria
- [ ] `mode: hold` correctly validates post-run observation windows (180s/300s) and continuous safeguard invariants.
- [ ] 1800s verification budget standard gives every converging check its full declared window (e.g. 120s) without premature truncation.
- [ ] Parallel child verdicts reached at the deadline are captured and recorded without data loss.
- [ ] Full suite of declarative verifiers (`http_probe`, `git_repo_sync`, `pod_exec`, `identity_preserved`, `subject_access_review`, `workspace_file`) passes unit and integration tests.
- [ ] Malformed or unparseable `verification_spec` entries fail fast before evaluation execution begins.

Contributor guide

Open the contributing guide

Research direction

Start with devops_bench/verification/hold.py and runner.py, then inspect the verifier implementations under devops_bench/verification/verifiers/. Run the referenced unit and integration tests and trace verification_spec parsing first; done means hold windows, fair budgets, deadline verdicts, the listed verifiers, and fail-fast error handling meet the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, kubernetes, python
Domain
devops, infrastructure, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.