kubernetes-sigs / kubernetes-sigs/devops-bench

eval: fix the defects that block the two migrated tasks from running and scoring

Open
#113 1 comment 0 reactions 1 assignee Claimed by @jessie1111101 View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

Both migrated tasks were unrunnable or produced untrustworthy scores. Found by running them end to end on GKE and kind. Owner: @jessie1111101.

**Cannot run at all**
- [ ] #64 add the `prebuilt/minimum` stack — `deploy-hello-app` names it but it never migrated
- [ ] #70 set an explicit provider on `deploy-hello-app` — fails one step later without it
- [ ] #69 fix the `opa-remediation` PolicyReport gate — read a field Kyverno 1.12 does not populate, so it could never pass at any timeout

**Scores were wrong, all in the inflating direction**
- [ ] #61 share the verification budget across converging entries — 5 entries consumed the whole 600s and the rest errored out of the score
- [ ] #74 treat `NotFound` as observed absence — a resource the agent never created was dropped from the denominator instead of counted
- [ ] #75 let a parallel child hand back a verdict it reached on time (fixes #67)

**Related**
- [ ] #43 load-fault transport ownership

**Verification.** With all of these applied, both tasks run end to end at `VerificationCoverage 1.0`: `deploy-hello-app` scores 1.0 and `opa-remediation` 0.8165 on gemini-3.1-pro-preview via openclaw. Before them, `opa-remediation` could not start and `deploy-hello-app` graded 4 of 13 objectives.

A combined branch exists for anyone who needs to run today: `integration/runnable` on `jessie1111101/devops-bench-upstream`. Scores from it will not reproduce on `main` until these merge.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.