kubernetes-sigs / kubernetes-sigs/devops-bench
Support benchmarking ADK-built agents
- Dominant language
- Python
- Stars
- 29
- Forks
- 15
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 15
Description
## Goal
Benchmark agents built with the [Agent Development Kit](https://google.github.io/adk-docs/), so ADK teams can run their agents against devops-bench tasks and get a scored trajectory back. Pathfinder is the first consumer.
**Constraint that shapes everything below:** the Pathfinder team cannot run against open source. Changes have to be mirrored into google3 and run there, so every gap we miss costs a full mirror round trip. The plan is therefore to batch, not to iterate.
## Done
- [x] #137 — `feat(agents): add an ADK agent harness`. Registers `--agent-type adk`. Imports the agent named by `AGENT_TARGET`, deep-copies it, applies MCP/skills/rules to **every** agent in the tree, drives it through ADK's `Runner`, folds the event stream into a trajectory.
- [x] #136 — `fix(deps): cap mcp below 2.0`.
## PRs needed
Ordered. 1 is the only one that silently corrupts results.
### 1. Fix output extraction for remote agents — #159 (open)
For an A2A agent, `AgentResult.output` comes back as the **last artifact's** text. The actual final answer sits in the task's `status.message` and is dropped. Confirmed with `use_legacy` both true and false, so it is consistent behaviour, not a legacy quirk.
Impact: the bench scores the agent against the wrong text. Any Pathfinder run before this lands is meaningless. Parser-only change, unit-tested from a recorded event. #159 also records a terminal task state other than completed on the result's errors, so a failed remote task is no longer scored as an answer.
### 2. Declare the `a2a` extra
Today the extra is bare `adk = ["google-adk>=2.7.1"]`. A2A needs `google-adk[a2a]` plus `a2a-sdk[grpc]` (grpcio + grpcio-tools). Packaging only.
### 3. Build the A2A client from config
So a caller gets a working gRPC connection from bench config instead of hand-writing a wrapper module — and, more importantly, without tripping the hazard in the "Validated" section below.
### 4. Sub-agent attribution in the trajectory
Two independent halves, both parser-only:
- **Native ADK trees** — ADK stamps `author` on every event and the parser discards it.
- **A2A agents** — per-sub-agent artifacts arrive in the event's `custom_metadata` and the parser ignores that field entirely.
This is FR2. See below for exactly how much of it comes free.
### 5. Smaller, unchanged from before
- Wire `AGENT_MAX_TURNS` to ADK's `RunConfig.max_llm_calls` (today the cap is ADK's default of 500).
- Support non-Gemini models — `LiteLlm` needs `google-adk[extensions]`, which the `adk` extra does not install. Not relevant to Pathfinder, which manages its own LLM calls.
- Map `AGENT_PROVIDER` / `AGENT_API_KEY` onto ADK's `GOOGLE_*` env. Currently a documented manual step.
- Token accounting for non-Gemini models.
- Persist harness metadata into `results.json`. Pre-existing; affects every harness, not just ADK.
## Validated
### The harness itself (#137)
On a real cluster, using throwaway probe agents so any failure was unambiguously the harness's fault:
- **Single agent** — `tasks/common/opa-remediation` on kind with a live `gke-mcp` toolset. 30 tool calls, `VerificationCoverage: 1.0`, `OutcomeScore: 0.8165`.
- **Multi-agent tree** — a three-node delegating tree. This found a real bug: capabilities were reaching the root agent only, so the sub-agent doing the cluster work had no tools. ADK resolves tools from whichever agent is active, with no inheritance from a parent. The broken run reported `status: success` with `errors: []` and scored `0.0` — a harness fault looked exactly like an agent failing the task. Fixed in #137.
- **Scoring parity** — the fixed tree then scored **exactly** what the single agent scored: `OutcomeScore 0.8165`, `VerificationCoverage 1.0`. A tree is graded like a single agent, with no harness penalty.
### FR1 — A2A over gRPC works today, with zero harness changes
Stood up a real A2A gRPC server (`GrpcHandler` + `grpc.aio`), pointed a `RemoteA2aAgent` at an AgentCard advertising the GRPC binding, and drove it through the merged harness. The response came back. `RemoteA2aAgent` is a `BaseAgent`, so the harness drives it unmodified. Multi-turn/`ContextId` is handled — round-tripped through `custom_metadata`.
Requirements: `a2a-sdk[grpc]` installed, an AgentCard advertising GRPC, and a `ClientFactory` configured with `supported_protocol_bindings=[GRPC]` and a `grpc_channel_factory`.
**Hazard, and the reason for PR 3.** You must *also* pass `httpx_client=` to `RemoteA2aAgent`. Otherwise `_ensure_httpx_client()` calls `_compat.rebind_client_factory_httpx`, which on a2a-sdk 1.x throws the caller's factory away and rebuilds it with `supported_protocol_bindings=[JSONRPC, HTTP_JSON]`. Its docstring states custom transports are *"not carried over — intended behavior."* The symptom is `no compatible transports found`, which points nowhere near the cause.
### FR2 — does not come free, but is tractable
Two runs through the merged harness, both Pathfinder-shaped:
| Tree shape | Raw ADK events | Parsed trajectory |
|---|---|---|
| LLM-driven delegation | `author` = coordinator, triage_agent, diagnostic_agent | 4 flat entries, no author. Order *is* recoverable from the `transfer_to_agent` entries |
| `SequentialAgent` | `author` = triage_agent, diagnostic_agent | 2 flat entries. **Zero** delegation markers |
Pathfinder is described as "6 internal sub-agents execute in sequence", which is the second row: the trajectory is indistinguishable from one agent making two tool calls.
For a **remote** agent the picture is different and better. `RemoteA2aAgent` stamps `author=self.name` on every converted event, so all sub-agents collapse into one name — but the artifacts survive intact in `custom_metadata`:
```
a2a:response.artifacts = [
{name: "triage_agent", metadata: {sub_agent: "triage_agent"}, parts: [...]},
{name: "diagnostic_agent", metadata: {sub_agent: "diagnostic_agent"}, parts: [...]}
]
```
So FR2 for Pathfinder is a parser change, not a protocol change — **provided Pathfinder emits one artifact per sub-agent.** See the open question below.
Note also: `ToolCall` is flat, so keeping `author` yields *attributed*, not *nested*. FR2 as written asks for nested.
### Scoring implications for a remote agent
`trajectory` is empty and all token counts are `None` — expected, since the agent runs its own LLM and exposes no tool calls. Scoring therefore rests entirely on the final output plus artifacts, which is what makes PR 1 and PR 4 load-bearing rather than nice to have.
## Open questions
1. **Does the real Pathfinder service emit one artifact per sub-agent, or a single report?** This decides whether PR 4 is small or turns into a Pathfinder-side change. → Pathfinder team
2. **Does google3 third_party carry `google-adk` ≥ 2.7.1, `a2a-sdk[grpc]`, and `mcp<2`?** If not, this blocks before any harness code matters. → whoever owns the mirror
3. **Is Pathfinder's AgentCard reachable, and does it advertise the GRPC binding?** → Pathfinder team
## Out of scope
- A trajectory that is genuinely *nested* rather than flat-with-attribution. Worth its own issue if the flat form proves insufficient.
- Endpoint-only tasks (no cluster). Separate task-type work.
## Docs
`docs/components/agents.md` covers harness selection, `AGENT_TARGET` spellings, deep-copy semantics, multi-agent tree behaviour, workspace/`cwd` behaviour, and the Vertex env recipe.
Contributor guide
Assessment
This issue has not been assessed yet.