google / google/adk-python

Proposal: Add built-in tool_trajectory_f1 evaluator

Đang mở
#5,306 1 bình luận 0 reaction 3 người được giao Được @i-yliu nhận Xem trên GitHub
eval needs review
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

## 🔴 Required Information

### Is your feature request related to a specific problem?

ADK's built-in trajectory evaluator (`tool_trajectory_in_order`) scores each invocation as binary pass/fail. This makes it hard to measure incremental improvements:

- an agent that calls 3 of 4 expected tools scores `0.0` — identical to an agent that called zero correct tools
- extra spurious tool calls collapse an otherwise correct trajectory to failure
- repeated tool names require one-to-one alignment, which greedy matching handles incorrectly

The result is a noisy signal that is hard to use for regression tracking during agent development.

### Describe the Solution You'd Like

A new built-in evaluator named `tool_trajectory_f1` that scores tool-call trajectories with partial credit using F1 scoring.

- **Metric name:** `tool_trajectory_f1`
- **Criterion type:** `ToolTrajectoryF1Criterion(BaseCriterion)`
- **Match modes:** `name_only`, `name_and_args`, `name_and_required_args`
- **Alignment:** ordered (monotonic) and unordered (maximum-cardinality)
- Returns `NOT_EVALUATED` when reference invocations are unavailable

Per-invocation scoring (A = actual count, E = expected count, M = matched pairs):

- `A == 0 and E == 0` → `1.0`
- `A == 0 xor E == 0` → `0.0`
- Otherwise: `precision = M/A`, `recall = M/E`, `f1 = 2·P·R / (P+R)`

Case score is the mean of per-invocation F1 scores.

### Impact on your work

I am building an evaluation harness for ADK-based agents and need a stable, deterministic regression metric for tool-use quality. The binary scorer makes it difficult to distinguish agents that are nearly correct from agents that are completely wrong, which slows down iterative improvement.

No specific timeline — but this would unblock more reliable eval-driven development for any team using ADK.

### Willingness to contribute

Yes. I am willing to implement this and submit a focused PR with unit tests, following the contribution guidelines (CLA signed).

---

## 🟡 Recommended Information

### Describe Alternatives You've Considered

- **Custom metric function via `custom_metrics`:** works today but requires per-project boilerplate, is not discoverable, and cannot be referenced by name in eval set configs the way built-in metrics can.
- **Patching `tool_trajectory_in_order`:** would break existing eval sets that rely on binary semantics.

### Proposed API / Implementation

```python
# eval_metrics.py
class ToolTrajectoryF1Criterion(BaseCriterion):
match_mode: Literal["name_only", "name_and_args", "name_and_required_args"] = "name_only"
ordered: bool = True

# trajectory_f1_evaluator.py
class ToolTrajectoryF1Evaluator(Evaluator):
criterion_type: ClassVar = ToolTrajectoryF1Criterion

async def evaluate_invocations(
self,
actual_invocations,
expected_invocations,
conversation_scenario=None,
) -> EvaluationResult:
...
```

Registration in `_get_default_metric_evaluator_registry()`:

```python
MetricInfo(
metric_name="tool_trajectory_f1",
description="F1 score for tool-call trajectory matching.",
),
```

Usage in an eval set config:

```json
{
"metric": "tool_trajectory_f1",
"criterion": { "threshold": 0.8, "matchMode": "name_only" }
}
```

### Additional Context

Issue #4794 proposes adding `ignore_args` to the existing trajectory evaluator. `tool_trajectory_f1` is complementary — it addresses partial-credit scoring rather than argument filtering, and could itself benefit from an `ignore_args`-style match mode once #4794 lands.

I'd appreciate early feedback on one API question: is a separate `ToolTrajectoryF1Criterion` preferred, or should `ToolTrajectoryCriterion` be extended with an optional `scoring: Literal["binary", "f1"]` field?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.