googleapis / googleapis/python-aiplatform

evals run_inference (local ADK agent): strips multimodal prompt parts (inline_data) before Runner.run_async

Đang mở
#6,976 0 bình luận 0 reaction 1 người được giao Được @jscud nhận Xem trên GitHub
api: vertex-ai
Ngôn ngữ chính
Python
Star
905
Fork
465
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
44

Mô tả

## Environment

- `google-cloud-aiplatform` 1.157.0
- `google-agents-cli` 0.4.0 (`agents-cli eval generate`)
- `google-adk` 2.2.0

## Summary

`vertexai.Client().evals.run_inference(agent=)` — the path used by `agents-cli eval generate` — **drops non-text prompt parts** before calling ADK `Runner.run_async`. Eval cases with multimodal `prompt.parts[]` (e.g. PDF `inline_data` per agents-cli multimodal eval schema) reach the agent as **text-only** user content. Tools that depend on `tool_context.user_content.parts[].inline_data` never see the attachment.

## Repro

1. Create a one-case eval dataset with multimodal prompt (text + PDF `inline_data`), e.g. agents-cli multimodal schema:

```json
{
"eval_cases": [{
"eval_case_id": "pdf_smoke",
"prompt": {
"role": "user",
"parts": [
{"text": "Process this document."},
{"inline_data": {"mime_type": "application/pdf", "data": ""}}
]
}
}]
}
```

2. Run local inference via agents-cli:

```bash
agents-cli eval generate --dataset --output traces.json
```

3. Inspect trace: agent tool that reads `user_content.parts` for `inline_data` returns an error like **"No document found"** — only the text part survived inference.

## Contrast (works)

ADK `AgentEvaluator` / pytest evaluation passes the full `Content` (text + `inline_data`) into `Runner.run_async`, and the same tool succeeds.

## Root cause

In `vertexai/_genai/_evals_common.py`:

### 1. DataFrame build flattens prompt to text

`_eval_cases_to_dataframe` stores:

```python
row[PROMPT] = _evals_data_converters._get_content_text(case.prompt)
```

`_get_content_text` (`_evals_data_converters.py`) concatenates `part.text` and **ignores** `inline_data`, `file_data`, etc.

### 2. Local agent run wraps string as single text Part

`_execute_local_agent_run_with_retry_async` calls:

```python
new_message_content = genai_types.Content(
role="user",
parts=[genai_types.Part(text=contents)],
)
```

So even if `contents` were a full `Content`, this stringifies it into one text part. Today `contents` is already the plain text string.

### 3. N+1 / `agent_data` path — same bug

When inference uses `agent_data` turns, the code still calls `_get_content_text(last_user_content)` before the local agent run.

## Expected behavior

Preserve full multimodal `Content` (all `parts`, including `inline_data` / `file_data`) through the local-agent inference path and call:

```python
Runner.run_async(new_message=Content.model_validate(full_content))
```

instead of `Part(text=...)`.

## Suggested fix locations

`vertexai/_genai/_evals_common.py`:

1. `_eval_cases_to_dataframe` — store full prompt `Content` dict (with `parts`) when multimodal, not `_get_content_text()` only.
2. `_extract_contents_for_inference` — if dict has `parts`, pass through (do not require nested `contents` key).
3. `_execute_local_agent_run_with_retry_async` — use `Content.model_validate(contents)` when `contents` is a Content dict; keep `Part(text=...)` fallback for plain strings.
4. N+1 `agent_data` branch — same helper instead of `_get_content_text(last_user_content)`.

## Impact

Blocks multimodal agent eval via `agents-cli eval generate` for any tool that reads uploaded bytes from `user_content` (PDF invoices, images, etc.). Workaround today is ADK `AgentEvaluator` pytest path, which preserves `inline_data`.

## References

- agents-cli multimodal eval schema: https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/references/multimodal-eval.md
- Related open eval issues: #6785

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.