GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

[Bug]: evaluating_adk_agent uses predicted_trajectory instead of reference_trajectory

Open
#2,370 0 comments 0 reactions 1 assignee Claimed by @inardini View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

### File Name

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluating_adk_agent.ipynb

### What happened?

In the Prepare Agent Evaluation dataset of the [evaluating_adk_agent](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluating_adk_agent.ipynb) notebook, there is a critical typo:
predicted_trajectory should be reference_trajectory

```
eval_data = {
"prompt": [
"Get price for smartphone",
"Get product details and price for headphones",
"Get details for usb charger",
"Get product details and price for shoes",
"Get product details for speaker?",
],
"predicted_trajectory": [. # should be reference_trajectory
[
{
"tool_name": "get_product_price",
"tool_input": {"product_name": "smartphone"},
}
...
```
compare it with the correct eval_data in this notebook [evaluating_langgraph_agent](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluating_langgraph_agent.ipynb):

```

eval_data = {
"prompt": [
"Get price for smartphone",
"Get product details and price for headphones",
"Get details for usb charger",
"Get product details and price for shoes",
"Get product details for speaker?",
],
"reference_trajectory": [
[
{
"tool_name": "get_product_price",
"tool_input": {"product_name": "smartphone"},
}
],

```
### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.