kubernetes-sigs / kubernetes-sigs/devops-bench
eval: fix the defects that block the two migrated tasks from running and scoring
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
Both migrated tasks were unrunnable or produced untrustworthy scores. Found by running them end to end on GKE and kind. Owner: @jessie1111101.
**Cannot run at all**
- [ ] #64 add the `prebuilt/minimum` stack — `deploy-hello-app` names it but it never migrated
- [ ] #70 set an explicit provider on `deploy-hello-app` — fails one step later without it
- [ ] #69 fix the `opa-remediation` PolicyReport gate — read a field Kyverno 1.12 does not populate, so it could never pass at any timeout
**Scores were wrong, all in the inflating direction**
- [ ] #61 share the verification budget across converging entries — 5 entries consumed the whole 600s and the rest errored out of the score
- [ ] #74 treat `NotFound` as observed absence — a resource the agent never created was dropped from the denominator instead of counted
- [ ] #75 let a parallel child hand back a verdict it reached on time (fixes #67)
**Related**
- [ ] #43 load-fault transport ownership
**Verification.** With all of these applied, both tasks run end to end at `VerificationCoverage 1.0`: `deploy-hello-app` scores 1.0 and `opa-remediation` 0.8165 on gemini-3.1-pro-preview via openclaw. Before them, `opa-remediation` could not start and `deploy-hello-app` graded 4 of 13 objectives.
A combined branch exists for anyone who needs to run today: `integration/runnable` on `jessie1111101/devops-bench-upstream`. Scores from it will not reproduce on `main` until these merge.
Contributor guide
Assessment
This issue has not been assessed yet.