kubernetes-sigs/devops-bench

View on GitHub

A standardized benchmarking suite to evaluate how well different agents or models perform specific DevOps tasks. Its goal is to provide an open-source, reproducible way to transparently assess agent performance across various infrastructure platforms and operational environments.

Stars: 30
Forks: 15
Open beginner issues: 0
Indexed issues: 37
Avg merge: 6d 3h
Merged PRs (30d): 15

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 18, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

37 open issues indexed

feat(results): Standardize universal token accounting, rejudge workflows, and offline results normalization Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#169 · Opened Sep 14, 2026
feat(verification): Enhance deterministic verification engine with hold-window monitoring, fair budget sharing, and expanded verifier suite Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#168 · Opened Sep 14, 2026
feat(agents): Harden agent CLI execution with flag forwarding, synchronized timeouts, multi-provider transports, and local serving Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

kubernetes-sigs/devops-bench#167 · Opened Sep 14, 2026
feat(infra): Implement fail-closed model preflight, campaign launcher contract, and robust teardown Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

kubernetes-sigs/devops-bench#166 · Opened Sep 14, 2026
A failed `tofu apply` reaches results.json without its stderr, so provisioning failures cannot be diagnosed after the run Open

Difficulty 4/5 3-5 days Newbie friendliness 67/100

kubernetes-sigs/devops-bench#157 · Opened Sep 10, 2026
Bastion scripts should not default the MCP server and skills to gke-mcp Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

kubernetes-sigs/devops-bench#140 · Opened Aug 28, 2026
Support benchmarking ADK-built agents Open

kubernetes-sigs/devops-bench#138 · 1 comment · 1 assignee · Opened Aug 25, 2026
feat(harness): Sanitize environment variables and breadcrumbs in subprocess execution Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kubernetes-sigs/devops-bench#134 · 1 comment · Opened Aug 25, 2026
feat(eval): Add post-run anti-cheating detection and transcript audit verifier Open

kubernetes-sigs/devops-bench#133 · 2 comments · 1 assignee · Opened Aug 25, 2026
feat(security): Implement workspace isolation and task definition hiding during agent execution Open

kubernetes-sigs/devops-bench#132 · 2 comments · 1 assignee · Opened Aug 25, 2026
Verifiers cannot see the agent's transcript: thread AgentResult into verification Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

kubernetes-sigs/devops-bench#118 · Opened Aug 20, 2026
Agent +skills source is empty: gke-mcp removed its skills/ directory upstream Open

kubernetes-sigs/devops-bench#117 · 1 comment · 1 assignee · Opened Aug 20, 2026
Promote k8s resource quantity parsing to a shared utility Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

kubernetes-sigs/devops-bench#116 · Opened Aug 20, 2026
feat(tasks): add four tasks with deterministic verification specs Open

kubernetes-sigs/devops-bench#114 · 1 comment · 1 assignee · Opened Aug 19, 2026
eval: fix the defects that block the two migrated tasks from running and scoring Open

kubernetes-sigs/devops-bench#113 · 1 comment · 1 assignee · Opened Aug 19, 2026
docs: migrate the docs and skills tail from gke-labs Open

kubernetes-sigs/devops-bench#112 · 2 comments · 1 assignee · Opened Aug 19, 2026
feat(infra): Add an EKS infra provider matching the existing provider interface for cross-cloud parity. Open

kubernetes-sigs/devops-bench#111 · 1 comment · 1 assignee · Opened Aug 19, 2026
Harden CVE remediation task with layered failure modes. Open

kubernetes-sigs/devops-bench#110 · 1 comment · 1 assignee · Opened Aug 19, 2026
feat(infra): Add an AKS infra provider matching the existing provider interface for cross-cloud parity. Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kubernetes-sigs/devops-bench#109 · Opened Aug 19, 2026
feat(tasks): Author complex cluster scalability & autoscaling task scenario Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 42/100

kubernetes-sigs/devops-bench#104 · 1 comment · Opened Aug 19, 2026
feat(tasks): Author complex Root Cause Analysis (RCA) task scenario Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#103 · 1 comment · Opened Aug 19, 2026
feat(eval): Add agentic dual-mode verifier supporting planned and autonomous verification Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#102 · 1 comment · Opened Aug 19, 2026
feat(eval): Support custom planned metrics in evaluator engine Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 38/100

kubernetes-sigs/devops-bench#101 · 1 comment · Opened Aug 19, 2026
feat(eval): Support planned declarative verification in tasks YAML schema Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#100 · 1 comment · Opened Aug 19, 2026
feat(leaderboard): Add leaderboard results integration for Hermes agent harness Open

kind/feature priority/important-soon

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kubernetes-sigs/devops-bench#99 · 1 comment · Opened Aug 19, 2026
feat(leaderboard): Expand leaderboard to support multi-model evaluation (Gemini, Claude, GPT) Open

kind/feature priority/important-soon

Difficulty 5/5 Over a week Newbie friendliness 35/100

kubernetes-sigs/devops-bench#98 · 1 comment · Opened Aug 19, 2026
feat(monitor): Add Performance Monitor to capture resource metrics across tasks Open

kind/feature priority/critical-urgent

kubernetes-sigs/devops-bench#97 · 3 comments · 1 assignee · Opened Aug 19, 2026
feat(eval): Implement automated prompt optimization loop for tasks Open

kind/feature priority/important-soon

Difficulty 5/5 Over a week Newbie friendliness 38/100

kubernetes-sigs/devops-bench#96 · 1 comment · Opened Aug 19, 2026
feat(infra): Add Bring-Your-Own (BYO) skills, tools, and MCP server support Open

kind/feature priority/important-soon

kubernetes-sigs/devops-bench#95 · 5 comments · 1 assignee · Opened Aug 19, 2026
feat(chaos): Extend Chaos agent with standardized fault injections and recovery metrics Open

kind/feature priority/important-soon

kubernetes-sigs/devops-bench#94 · 2 comments · 1 assignee · Opened Aug 19, 2026
feat(infra): Implement soft and hard reconciliation reset for test clusters Open

kind/feature priority/important-longterm

Difficulty 5/5 Over a week Newbie friendliness 25/100

kubernetes-sigs/devops-bench#93 · 1 comment · Opened Aug 19, 2026
feat(eval): Implement Pass@5 and Pass^5 evaluation metrics Open

kind/feature priority/important-soon

kubernetes-sigs/devops-bench#92 · 2 comments · 1 assignee · Opened Aug 19, 2026
feat(infra): Add vCluster support to Infra Provisioner Open

kind/feature priority/important-soon

kubernetes-sigs/devops-bench#91 · 2 comments · 1 assignee · Opened Aug 19, 2026
Parallel verification discards a child's real result when it lands at the shared deadline Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

kubernetes-sigs/devops-bench#67 · Opened Jul 31, 2026
Add git_repo_sync verification for the OPA remediation GitOps objective Open

kubernetes-sigs/devops-bench#56 · 1 comment · 1 assignee · Opened Jul 30, 2026
chaos: construct the fortio load argv from structured parameters instead of a model-composed command Open

kind/cleanup

kubernetes-sigs/devops-bench#33 · 1 assignee · Opened Jul 18, 2026
Determine what type of eval framework we should use Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

kubernetes-sigs/devops-bench#3 · 3 comments · 2 reactions · Opened Jun 8, 2026

kubernetes-sigs/devops-bench

Get new issues in your inbox