google / google/adk-python

Proposal: Add built-in tool_parameter_match evaluator

Abierto
#5,643 1 comentario 0 reacciones 3 asignados Reclamado por @i-yliu Ver en GitHub
eval needs review
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

## 🔴 Required Information

### Is your feature request related to a specific problem?

ADK does not currently expose a dedicated built-in metric for tool argument quality. Existing trajectory metrics tell you whether a tool was called, but not whether the agent supplied the right parameters. This hides important failure modes:

- correct tool name with incorrect required arguments
- partial argument correctness — values that are close but not exact
- repeated same-tool calls that require stable one-to-one alignment between expected and actual invocations before argument-level scoring is meaningful

For regression tracking during agent development, "the right tool was called" is necessary but not sufficient. Without argument-level scoring, agents that systematically miscall a correctly-chosen tool produce no signal in built-in metrics.

### Describe the Solution You'd Like

A new built-in evaluator named `tool_parameter_match` that scores the quality of tool-call arguments after deterministic alignment between expected and actual invocations.

- **Metric name:** `tool_parameter_match`
- **Criterion type:** `ToolParameterMatchCriterion(BaseCriterion)`
- **Match modes:** `name_only`, `name_and_args`, `name_and_required_args` (alignment behavior, mirroring trajectory scoring)
- **Argument strategies:** `exact`, `casefold_exact`, `numeric`, `contains`
- **Per-argument strategy override:** optional `per_arg_strategies` dict
- **Numeric tolerance:** optional `numeric_tolerance` for the `numeric` strategy
- Returns `NOT_EVALUATED` when reference invocations are unavailable
- Returns per-invocation `NOT_EVALUATED` when an invocation has zero expected tool calls

Per-call scoring (for each matched expected tool call):

- score the expected argument keys only
- average the per-argument match scores
- if the expected argument dict is empty, the tool-call score is `1.0`

For unmatched expected calls: score `0.0`.

Case score is the mean of evaluated per-invocation scores, excluding `NOT_EVALUATED` invocations.

### Impact on your work

I am building an evaluation harness for ADK-based agents and need a deterministic, regression-friendly metric for argument-level correctness. Without one, regressions where the agent picks the right tool but supplies wrong or partial arguments are invisible to built-in scoring, which slows down iterative improvement.

No specific timeline — but this would unblock more reliable eval-driven development for any team using ADK.

### Willingness to contribute

Yes. I am willing to implement this and submit a focused PR with unit tests, following the contribution guidelines (CLA signed).

---

## 🟡 Recommended Information

### Describe Alternatives You've Considered

- **Custom metric function via `custom_metrics`:** works today but requires per-project boilerplate, is not discoverable, and cannot be referenced by name in eval-set configs the way built-in metrics can.
- **Embedding argument matching inside `tool_trajectory_in_order`:** would conflate trajectory shape with argument quality and break existing eval sets that rely on the trajectory metric's current binary semantics.

### Proposed API / Implementation

```python
# eval_metrics.py
ArgumentStrategy = Literal["exact", "casefold_exact", "numeric", "contains"]

class ToolParameterMatchCriterion(BaseCriterion):
match_mode: Literal["name_only", "name_and_args", "name_and_required_args"] = "name_and_required_args"
default_strategy: ArgumentStrategy = "exact"
per_arg_strategies: dict[str, ArgumentStrategy] | None = None
numeric_tolerance: float = 0.0
ordered: bool = True

# parameter_match_evaluator.py
class ToolParameterMatchEvaluator(Evaluator):
criterion_type: ClassVar = ToolParameterMatchCriterion

async def evaluate_invocations(
self,
actual_invocations,
expected_invocations,
conversation_scenario=None,
) -> EvaluationResult:
...
```

Registration in `_get_default_metric_evaluator_registry()`:

```python
MetricInfo(
metric_name="tool_parameter_match",
description="Argument-level match score for deterministically aligned tool calls.",
),
```

Usage in an eval set config:

```json
{
"metric": "tool_parameter_match",
"criterion": {
"threshold": 0.8,
"matchMode": "name_and_required_args",
"defaultStrategy": "exact",
"perArgStrategies": { "temperature": "numeric" },
"numericTolerance": 0.5
}
}
```

### Additional Context

Filed alongside #5306 (`tool_trajectory_f1`). The two metrics are independent but address the same workflow gap — partial-credit, deterministic tool-use scoring for regression tracking. They are designed to compose: trajectory scoring decides whether the right tools were called; parameter scoring decides whether each call's arguments were correct.

Issue #4794 proposes adding `ignore_args` to the existing trajectory evaluator. `tool_parameter_match` is complementary — it provides positive argument-quality scoring rather than argument filtering.

I'd appreciate early feedback on two API questions:

1. Is a separate `ToolParameterMatchCriterion` preferred, or should argument-quality scoring be folded into an existing argument-aware criterion?
2. Should call-alignment logic be shared with `tool_trajectory_in_order` / `tool_trajectory_f1` via a common helper, or recomputed inside each metric independently?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.