kubernetes-sigs / kubernetes-sigs/devops-bench
feat(infra): Add Bring-Your-Own (BYO) skills, tools, and MCP server support
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
### Problem / Motivation
Different autonomous agents and research teams use specialized CLI tools, internal tool suites, or Model Context Protocol (MCP) servers. The benchmark infrastructure should be modular and allow any external agent harness to plug in its own custom tools and skills without modifying core benchmark code.
### Proposed Solution & Technical Details
- Implement a pluggable tool / skill registry in the benchmark runner.
- Support Model Context Protocol (MCP) server lifecycle management (starting local MCP server processes, passing MCP connection configs to agent runners).
- Enable custom environment variable injection and skill definition files per evaluation configuration.
### Acceptance Criteria
- [ ] Agent runners can declare external MCP server endpoints or launch configs in benchmark run specs.
- [ ] Benchmark runner successfully provisions and passes MCP tools to test agent environments.
- [ ] Example documentation demonstrating how to benchmark an agent with custom MCP tools.
---
/kind feature
/area infra
/priority important-soon
Contributor guide
Assessment
This issue has not been assessed yet.