kubernetes-sigs / kubernetes-sigs/devops-bench
feat(leaderboard): Add leaderboard results integration for Hermes agent harness
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Problem / Motivation
To enable cross-agent comparisons on the public leaderboard, we need standard ingestion pipelines and result adapters for the Hermes autonomous agent harness.
### Proposed Solution & Technical Details
- Implement an evaluation adapter that parses Hermes agent execution traces and evaluation outputs.
- Normalize Hermes scoring results into the standard DevOps Bench result schema.
- Add Hermes harness entries to the leaderboard configuration and comparative visualization tables.
### Acceptance Criteria
- [ ] Parser successfully converts Hermes execution outputs to standard benchmark score JSON.
- [ ] Benchmark results from Hermes runs are rendered correctly in the leaderboard.
---
/kind feature
/area leaderboard
/priority important-soon
Contributor guide
Research direction
Locate the existing evaluation adapters, leaderboard configuration, and comparative visualization tables, then inspect how current harness outputs become standard benchmark score JSON. Start with the existing leaderboard or benchmark checks, if present, and verify that Hermes results render correctly and satisfy both acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json, python
- Domain
- data, data-visualization, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100