GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

[Bug]: Multi turn conversation eval metrics issue

Open
#2,979 0 comments 0 reactions 1 assignee Claimed by @coolalexzb View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

### File Name

multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb

### What happened?

A bug happened!
WHen running this notebook: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb

Multi turn metrics are not being defined both locally and on cloud.

Loss Analysis fails completely both locally and on cloud.

Log:
Evaluation job completed in 11 seconds.
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_861893970874073088.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_8902508155590672384.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4682072354791096320.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_2833907657708929024.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4750189299155075072.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
Status: FAILED

Error:

code=9 details

### Relevant log output

```shell

```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.