googleapis / googleapis/python-aiplatform

evals: ConversationTurn rejects extra fields from run_inference API response and missing turn_index

Đang mở
#6,785 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: vertex-ai
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

## Environment

- `google-cloud-aiplatform`: 1.152.0
- `google-genai`: 1.75.0
- `google-adk`: 1.33.0
- Python: 3.12
- Region: us-central1

## Description

When using the simulated evaluation pipeline (`generate_conversation_scenarios` → `run_inference` → `evaluate`), two bugs prevent end-to-end execution:

### Bug 1: ConversationTurn rejects extra fields from API response

`run_inference()` returns agent turn data with fields (`model_version`, `content`, `id`, `timestamp`, `author`, `actions`, `invocation_id`, `long_running_tool_ids`, `finish_reason`, `usage_metadata`, `avg_logprobs`) that are not defined in `ConversationTurn` (which only has `turn_index`, `turn_id`, `events`).

Since `google.genai._common.BaseModel` sets `extra='forbid'`, pydantic rejects these:

```
pydantic_core._pydantic_core.ValidationError: 44 validation errors for AgentData
turns.0.model_version
Extra inputs are not permitted
turns.0.content
Extra inputs are not permitted
...
```

The error originates at `_evals_common.py:1880` in `_process_multi_turn_agent_response`:
```python
return types.evals.AgentData(
turns=resp_item,
agents=agent_data_agents,
).model_dump(exclude_unset=True)
```

**Note:** The SDK has `_remove_extra_fields()` in `google/genai/_common.py:317` that handles this for `_from_response()` calls, but `_process_multi_turn_agent_response` uses direct construction instead.

### Bug 2: turn_index not populated by run_inference

The raw turn dicts from the agent engine response don't include `turn_index`. `_process_multi_turn_agent_response` passes them directly to `AgentData(turns=...)` without adding `turn_index`. When the resulting data is sent to the `evaluate` API, it fails:

```
400 INVALID_ARGUMENT: Field: instance.agent_eval_data.turns[0].turn_index; Message: Required field is not set.
```

### Workaround

We patched both issues in our eval script:

```python
# Bug 1: Set ConversationTurn to ignore unknown fields
ct = evals_types.ConversationTurn
ct.model_config["extra"] = "ignore"
ct.__pydantic_complete__ = False
ct.model_rebuild(force=True)
evals_types.AgentData.__pydantic_complete__ = False
evals_types.AgentData.model_rebuild(force=True)

# Bug 2: Inject turn_index based on position
_orig_process = _evals_common._process_multi_turn_agent_response
def _patched_process(resp_item, agent_data_agents):
if isinstance(resp_item, list):
for i, turn in enumerate(resp_item):
if isinstance(turn, dict) and "turn_index" not in turn:
turn["turn_index"] = i
return _orig_process(resp_item, agent_data_agents)
_evals_common._process_multi_turn_agent_response = _patched_process
```

### Steps to Reproduce

```python
import vertexai
from vertexai import Client, types

vertexai.init(project="PROJECT", location="us-central1")
client = Client(project="PROJECT", location="us-central1")

# Generate scenarios
eval_dataset = client.evals.generate_conversation_scenarios(
agent_info=agent_info,
config={"count": 1, "generation_instruction": "Search for hotels"},
allow_cross_region_model=True,
)

# This fails with Bug 1
eval_dataset_with_traces = client.evals.run_inference(
agent="projects/PROJECT_NUM/locations/us-central1/reasoningEngines/ENGINE_ID",
src=eval_dataset,
config={"user_simulator_config": {"max_turn": 3, "model_name": "gemini-2.5-flash"}},
)

# If Bug 1 is patched, this fails with Bug 2
eval_result = client.evals.evaluate(
dataset=eval_dataset_with_traces,
metrics=[types.RubricMetric.FINAL_RESPONSE_QUALITY],
)
```

### Suggested Fix

1. Add the missing fields to `ConversationTurn` in `vertexai/_genai/types/evals.py`, or set `extra='ignore'` on the model
2. In `_process_multi_turn_agent_response`, add `turn["turn_index"] = i` for each turn before constructing `AgentData`

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.