kubernetes-sigs / kubernetes-sigs/devops-bench

feat(eval): Add post-run anti-cheating detection and transcript audit verifier

Open
#133 2 comments 0 reactions 1 assignee Claimed by @isadominguez314 View on GitHub
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Summary
Add an automated transcript verifier pass that audits execution logs (`AgentResult`) for unauthorized tool invocations or shell commands attempting to read test definitions or bypass evaluation.

### Scope
1. Inspect tool calls and command history for access to task files, `task.yaml`, or benchmark internals.
2. Automatically flag and invalidate tainted benchmark runs.

Ref: Internal bug b/552484220

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.