googleapis / googleapis/python-aiplatform

evals: ConversationTurn rejects extra fields from run_inference API response and missing turn_index

Abierto
#6,785 0 comentarios 0 reacciones 0 asignados Ver en GitHub
api: vertex-ai
Lenguaje dominante
Python
Estrellas
905
Forks
465
Merge medio
1 d 13 h
PR fusionados (30 d)
44

Descripción

## Environment

- `google-cloud-aiplatform`: 1.152.0
- `google-genai`: 1.75.0
- `google-adk`: 1.33.0
- Python: 3.12
- Region: us-central1

## Description

When using the simulated evaluation pipeline (`generate_conversation_scenarios` → `run_inference` → `evaluate`), two bugs prevent end-to-end execution:

### Bug 1: ConversationTurn rejects extra fields from API response

`run_inference()` returns agent turn data with fields (`model_version`, `content`, `id`, `timestamp`, `author`, `actions`, `invocation_id`, `long_running_tool_ids`, `finish_reason`, `usage_metadata`, `avg_logprobs`) that are not defined in `ConversationTurn` (which only has `turn_index`, `turn_id`, `events`).

Since `google.genai._common.BaseModel` sets `extra='forbid'`, pydantic rejects these:

```
pydantic_core._pydantic_core.ValidationError: 44 validation errors for AgentData
turns.0.model_version
Extra inputs are not permitted
turns.0.content
Extra inputs are not permitted
...
```

The error originates at `_evals_common.py:1880` in `_process_multi_turn_agent_response`:
```python
return types.evals.AgentData(
turns=resp_item,
agents=agent_data_agents,
).model_dump(exclude_unset=True)
```

**Note:** The SDK has `_remove_extra_fields()` in `google/genai/_common.py:317` that handles this for `_from_response()` calls, but `_process_multi_turn_agent_response` uses direct construction instead.

### Bug 2: turn_index not populated by run_inference

The raw turn dicts from the agent engine response don't include `turn_index`. `_process_multi_turn_agent_response` passes them directly to `AgentData(turns=...)` without adding `turn_index`. When the resulting data is sent to the `evaluate` API, it fails:

```
400 INVALID_ARGUMENT: Field: instance.agent_eval_data.turns[0].turn_index; Message: Required field is not set.
```

### Workaround

We patched both issues in our eval script:

```python
# Bug 1: Set ConversationTurn to ignore unknown fields
ct = evals_types.ConversationTurn
ct.model_config["extra"] = "ignore"
ct.__pydantic_complete__ = False
ct.model_rebuild(force=True)
evals_types.AgentData.__pydantic_complete__ = False
evals_types.AgentData.model_rebuild(force=True)

# Bug 2: Inject turn_index based on position
_orig_process = _evals_common._process_multi_turn_agent_response
def _patched_process(resp_item, agent_data_agents):
if isinstance(resp_item, list):
for i, turn in enumerate(resp_item):
if isinstance(turn, dict) and "turn_index" not in turn:
turn["turn_index"] = i
return _orig_process(resp_item, agent_data_agents)
_evals_common._process_multi_turn_agent_response = _patched_process
```

### Steps to Reproduce

```python
import vertexai
from vertexai import Client, types

vertexai.init(project="PROJECT", location="us-central1")
client = Client(project="PROJECT", location="us-central1")

# Generate scenarios
eval_dataset = client.evals.generate_conversation_scenarios(
agent_info=agent_info,
config={"count": 1, "generation_instruction": "Search for hotels"},
allow_cross_region_model=True,
)

# This fails with Bug 1
eval_dataset_with_traces = client.evals.run_inference(
agent="projects/PROJECT_NUM/locations/us-central1/reasoningEngines/ENGINE_ID",
src=eval_dataset,
config={"user_simulator_config": {"max_turn": 3, "model_name": "gemini-2.5-flash"}},
)

# If Bug 1 is patched, this fails with Bug 2
eval_result = client.evals.evaluate(
dataset=eval_dataset_with_traces,
metrics=[types.RubricMetric.FINAL_RESPONSE_QUALITY],
)
```

### Suggested Fix

1. Add the missing fields to `ConversationTurn` in `vertexai/_genai/types/evals.py`, or set `extra='ignore'` on the model
2. In `_process_multi_turn_agent_response`, add `turn["turn_index"] = i` for each turn before constructing `AgentData`

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.