kubernetes-sigs / kubernetes-sigs/devops-bench

Verifiers cannot see the agent's transcript: thread AgentResult into verification

Open
#118 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

BaseVerifier.verify() takes only timeout_sec, and every built-in verifier reads cluster state — so a verifier that checks the agent's final response text or tool trace has nothing to read. The data exists: AgentResult.output and AgentResult.trajectory are in scope at run_task's verification call site (evalharness/default.py: execute_agent at ~803, _run_verification at ~827), but _run_verification passes only (entry, timeout_sec) down to the leaves, and the result record is written after verification runs.

Downstream we (gke-labs/kube-agents) worked around it: our AgentHarness.run() stashes output+trajectory in module state, and our entry-point verifiers (report_contains, tool_called) read the stash. It works because task execution is single-threaded, but it has a real residual: on the exception path where run_task still verifies although the harness raised before execute_agent, the stash holds the previous task's transcript. Fail-closed guards contain it; threading the data properly removes it.

Proposal: pass agent_res into _run_verification and set it on verifiers that declare an opt-in field (e.g. a wants_transcript marker or a typed context attribute), keeping verify(timeout_sec) source-compatible for existing verifiers. Happy to send a PR if the shape sounds right.

Written with the help of Claude as part of kube-agents' testing work; pinned SHA in our deployment is 4670d76.

Contributor guide

Open the contributing guide

Research direction

Start in evalharness/default.py at execute_agent around line 803 and _run_verification around line 827, then trace BaseVerifier.verify() and AgentResult.output and AgentResult.trajectory. Define and implement the opt-in transcript path without breaking existing verify(timeout_sec) callers. Done means eligible verifiers can inspect the current agent transcript, including exception-path handling, while existing verifiers remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.