kubernetes-sigs / kubernetes-sigs/devops-bench
feat(eval): Add post-run anti-cheating detection and transcript audit verifier
Open
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Summary
Add an automated transcript verifier pass that audits execution logs (`AgentResult`) for unauthorized tool invocations or shell commands attempting to read test definitions or bypass evaluation.
### Scope
1. Inspect tool calls and command history for access to task files, `task.yaml`, or benchmark internals.
2. Automatically flag and invalidate tainted benchmark runs.
Ref: Internal bug b/552484220
Contributor guide
Assessment
This issue has not been assessed yet.