googleapis / googleapis/python-aiplatform
evals: ConversationTurn rejects extra fields from run_inference API response and missing turn_index
- Langage dominant
- Python
- Étoiles
- 905
- Forks
- 465
- Merge moyen
- 1 j 13 h
- PR mergées (30 j)
- 44
Description
## Environment
- `google-cloud-aiplatform`: 1.152.0
- `google-genai`: 1.75.0
- `google-adk`: 1.33.0
- Python: 3.12
- Region: us-central1
## Description
When using the simulated evaluation pipeline (`generate_conversation_scenarios` → `run_inference` → `evaluate`), two bugs prevent end-to-end execution:
### Bug 1: ConversationTurn rejects extra fields from API response
`run_inference()` returns agent turn data with fields (`model_version`, `content`, `id`, `timestamp`, `author`, `actions`, `invocation_id`, `long_running_tool_ids`, `finish_reason`, `usage_metadata`, `avg_logprobs`) that are not defined in `ConversationTurn` (which only has `turn_index`, `turn_id`, `events`).
Since `google.genai._common.BaseModel` sets `extra='forbid'`, pydantic rejects these:
```
pydantic_core._pydantic_core.ValidationError: 44 validation errors for AgentData
turns.0.model_version
Extra inputs are not permitted
turns.0.content
Extra inputs are not permitted
...
```
The error originates at `_evals_common.py:1880` in `_process_multi_turn_agent_response`:
```python
return types.evals.AgentData(
turns=resp_item,
agents=agent_data_agents,
).model_dump(exclude_unset=True)
```
**Note:** The SDK has `_remove_extra_fields()` in `google/genai/_common.py:317` that handles this for `_from_response()` calls, but `_process_multi_turn_agent_response` uses direct construction instead.
### Bug 2: turn_index not populated by run_inference
The raw turn dicts from the agent engine response don't include `turn_index`. `_process_multi_turn_agent_response` passes them directly to `AgentData(turns=...)` without adding `turn_index`. When the resulting data is sent to the `evaluate` API, it fails:
```
400 INVALID_ARGUMENT: Field: instance.agent_eval_data.turns[0].turn_index; Message: Required field is not set.
```
### Workaround
We patched both issues in our eval script:
```python
# Bug 1: Set ConversationTurn to ignore unknown fields
ct = evals_types.ConversationTurn
ct.model_config["extra"] = "ignore"
ct.__pydantic_complete__ = False
ct.model_rebuild(force=True)
evals_types.AgentData.__pydantic_complete__ = False
evals_types.AgentData.model_rebuild(force=True)
# Bug 2: Inject turn_index based on position
_orig_process = _evals_common._process_multi_turn_agent_response
def _patched_process(resp_item, agent_data_agents):
if isinstance(resp_item, list):
for i, turn in enumerate(resp_item):
if isinstance(turn, dict) and "turn_index" not in turn:
turn["turn_index"] = i
return _orig_process(resp_item, agent_data_agents)
_evals_common._process_multi_turn_agent_response = _patched_process
```
### Steps to Reproduce
```python
import vertexai
from vertexai import Client, types
vertexai.init(project="PROJECT", location="us-central1")
client = Client(project="PROJECT", location="us-central1")
# Generate scenarios
eval_dataset = client.evals.generate_conversation_scenarios(
agent_info=agent_info,
config={"count": 1, "generation_instruction": "Search for hotels"},
allow_cross_region_model=True,
)
# This fails with Bug 1
eval_dataset_with_traces = client.evals.run_inference(
agent="projects/PROJECT_NUM/locations/us-central1/reasoningEngines/ENGINE_ID",
src=eval_dataset,
config={"user_simulator_config": {"max_turn": 3, "model_name": "gemini-2.5-flash"}},
)
# If Bug 1 is patched, this fails with Bug 2
eval_result = client.evals.evaluate(
dataset=eval_dataset_with_traces,
metrics=[types.RubricMetric.FINAL_RESPONSE_QUALITY],
)
```
### Suggested Fix
1. Add the missing fields to `ConversationTurn` in `vertexai/_genai/types/evals.py`, or set `extra='ignore'` on the model
2. In `_process_multi_turn_agent_response`, add `turn["turn_index"] = i` for each turn before constructing `AgentData`
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.