kubernetes-sigs / kubernetes-sigs/devops-bench

feat(leaderboard): Add leaderboard results integration for Hermes agent harness

Open
#99 1 comment 0 reactions 0 assignees View on GitHub
kind/feature priority/important-soon
Dominant language
Python
Stars
29
Forks
15
Avg merge
6d 3h
Merged PRs (30d)
15

Description

### Problem / Motivation
To enable cross-agent comparisons on the public leaderboard, we need standard ingestion pipelines and result adapters for the Hermes autonomous agent harness.

### Proposed Solution & Technical Details
- Implement an evaluation adapter that parses Hermes agent execution traces and evaluation outputs.
- Normalize Hermes scoring results into the standard DevOps Bench result schema.
- Add Hermes harness entries to the leaderboard configuration and comparative visualization tables.

### Acceptance Criteria
- [ ] Parser successfully converts Hermes execution outputs to standard benchmark score JSON.
- [ ] Benchmark results from Hermes runs are rendered correctly in the leaderboard.

---
/kind feature
/area leaderboard
/priority important-soon

Contributor guide

Open the contributing guide

Research direction

Locate the existing evaluation adapters, leaderboard configuration, and comparative visualization tables, then inspect how current harness outputs become standard benchmark score JSON. Start with the existing leaderboard or benchmark checks, if present, and verify that Hermes results render correctly and satisfy both acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, python
Domain
data, data-visualization, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.