googleapis / googleapis/python-aiplatform
evals run_inference (local ADK agent): strips multimodal prompt parts (inline_data) before Runner.run_async
- Vorherrschende Sprache
- Python
- Sterne
- 905
- Forks
- 465
- Ø Merge
- 1 T. 13 Std.
- Gemergte PRs (30 T.)
- 44
Beschreibung
## Environment
- `google-cloud-aiplatform` 1.157.0
- `google-agents-cli` 0.4.0 (`agents-cli eval generate`)
- `google-adk` 2.2.0
## Summary
`vertexai.Client().evals.run_inference(agent=)` — the path used by `agents-cli eval generate` — **drops non-text prompt parts** before calling ADK `Runner.run_async`. Eval cases with multimodal `prompt.parts[]` (e.g. PDF `inline_data` per agents-cli multimodal eval schema) reach the agent as **text-only** user content. Tools that depend on `tool_context.user_content.parts[].inline_data` never see the attachment.
## Repro
1. Create a one-case eval dataset with multimodal prompt (text + PDF `inline_data`), e.g. agents-cli multimodal schema:
```json
{
"eval_cases": [{
"eval_case_id": "pdf_smoke",
"prompt": {
"role": "user",
"parts": [
{"text": "Process this document."},
{"inline_data": {"mime_type": "application/pdf", "data": ""}}
]
}
}]
}
```
2. Run local inference via agents-cli:
```bash
agents-cli eval generate --dataset --output traces.json
```
3. Inspect trace: agent tool that reads `user_content.parts` for `inline_data` returns an error like **"No document found"** — only the text part survived inference.
## Contrast (works)
ADK `AgentEvaluator` / pytest evaluation passes the full `Content` (text + `inline_data`) into `Runner.run_async`, and the same tool succeeds.
## Root cause
In `vertexai/_genai/_evals_common.py`:
### 1. DataFrame build flattens prompt to text
`_eval_cases_to_dataframe` stores:
```python
row[PROMPT] = _evals_data_converters._get_content_text(case.prompt)
```
`_get_content_text` (`_evals_data_converters.py`) concatenates `part.text` and **ignores** `inline_data`, `file_data`, etc.
### 2. Local agent run wraps string as single text Part
`_execute_local_agent_run_with_retry_async` calls:
```python
new_message_content = genai_types.Content(
role="user",
parts=[genai_types.Part(text=contents)],
)
```
So even if `contents` were a full `Content`, this stringifies it into one text part. Today `contents` is already the plain text string.
### 3. N+1 / `agent_data` path — same bug
When inference uses `agent_data` turns, the code still calls `_get_content_text(last_user_content)` before the local agent run.
## Expected behavior
Preserve full multimodal `Content` (all `parts`, including `inline_data` / `file_data`) through the local-agent inference path and call:
```python
Runner.run_async(new_message=Content.model_validate(full_content))
```
instead of `Part(text=...)`.
## Suggested fix locations
`vertexai/_genai/_evals_common.py`:
1. `_eval_cases_to_dataframe` — store full prompt `Content` dict (with `parts`) when multimodal, not `_get_content_text()` only.
2. `_extract_contents_for_inference` — if dict has `parts`, pass through (do not require nested `contents` key).
3. `_execute_local_agent_run_with_retry_async` — use `Content.model_validate(contents)` when `contents` is a Content dict; keep `Part(text=...)` fallback for plain strings.
4. N+1 `agent_data` branch — same helper instead of `_get_content_text(last_user_content)`.
## Impact
Blocks multimodal agent eval via `agents-cli eval generate` for any tool that reads uploaded bytes from `user_content` (PDF invoices, images, etc.). Workaround today is ADK `AgentEvaluator` pytest path, which preserves `inline_data`.
## References
- agents-cli multimodal eval schema: https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/references/multimodal-eval.md
- Related open eval issues: #6785
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.