GoogleCloudPlatform / GoogleCloudPlatform/generative-ai
[Bug]: Multi turn conversation eval metrics issue
- Dominant language
- Jupyter Notebook
- Stars
- 17.7k
- Forks
- 4.5k
- Avg merge
- 12h 38m
- Merged PRs (30d)
- 42
Description
### File Name
multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
### What happened?
A bug happened!
WHen running this notebook: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
Multi turn metrics are not being defined both locally and on cloud.
Loss Analysis fails completely both locally and on cloud.
Log:
Evaluation job completed in 11 seconds.
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_861893970874073088.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_8902508155590672384.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4682072354791096320.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_2833907657708929024.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4750189299155075072.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
Status: FAILED
Error:
code=9 details
### Relevant log output
```shell
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.