GoogleCloudPlatform / GoogleCloudPlatform/generative-ai
[Bug]: evaluating_adk_agent uses predicted_trajectory instead of reference_trajectory
- Dominant language
- Jupyter Notebook
- Stars
- 17.7k
- Forks
- 4.5k
- Avg merge
- 12h 38m
- Merged PRs (30d)
- 42
Description
### File Name
https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluating_adk_agent.ipynb
### What happened?
In the Prepare Agent Evaluation dataset of the [evaluating_adk_agent](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluating_adk_agent.ipynb) notebook, there is a critical typo:
predicted_trajectory should be reference_trajectory
```
eval_data = {
"prompt": [
"Get price for smartphone",
"Get product details and price for headphones",
"Get details for usb charger",
"Get product details and price for shoes",
"Get product details for speaker?",
],
"predicted_trajectory": [. # should be reference_trajectory
[
{
"tool_name": "get_product_price",
"tool_input": {"product_name": "smartphone"},
}
...
```
compare it with the correct eval_data in this notebook [evaluating_langgraph_agent](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluating_langgraph_agent.ipynb):
```
eval_data = {
"prompt": [
"Get price for smartphone",
"Get product details and price for headphones",
"Get details for usb charger",
"Get product details and price for shoes",
"Get product details for speaker?",
],
"reference_trajectory": [
[
{
"tool_name": "get_product_price",
"tool_input": {"product_name": "smartphone"},
}
],
```
### Relevant log output
```shell
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.