googleapis / googleapis/python-aiplatform

evals run_inference (local ADK agent): strips multimodal prompt parts (inline_data) before Runner.run_async

Abierto
#6,976 0 comentarios 0 reacciones 1 asignado Reclamado por @jscud Ver en GitHub
api: vertex-ai
Lenguaje dominante
Python
Estrellas
905
Forks
465
Merge medio
1 d 13 h
PR fusionados (30 d)
44

Descripción

## Environment

- `google-cloud-aiplatform` 1.157.0
- `google-agents-cli` 0.4.0 (`agents-cli eval generate`)
- `google-adk` 2.2.0

## Summary

`vertexai.Client().evals.run_inference(agent=)` — the path used by `agents-cli eval generate` — **drops non-text prompt parts** before calling ADK `Runner.run_async`. Eval cases with multimodal `prompt.parts[]` (e.g. PDF `inline_data` per agents-cli multimodal eval schema) reach the agent as **text-only** user content. Tools that depend on `tool_context.user_content.parts[].inline_data` never see the attachment.

## Repro

1. Create a one-case eval dataset with multimodal prompt (text + PDF `inline_data`), e.g. agents-cli multimodal schema:

```json
{
"eval_cases": [{
"eval_case_id": "pdf_smoke",
"prompt": {
"role": "user",
"parts": [
{"text": "Process this document."},
{"inline_data": {"mime_type": "application/pdf", "data": ""}}
]
}
}]
}
```

2. Run local inference via agents-cli:

```bash
agents-cli eval generate --dataset --output traces.json
```

3. Inspect trace: agent tool that reads `user_content.parts` for `inline_data` returns an error like **"No document found"** — only the text part survived inference.

## Contrast (works)

ADK `AgentEvaluator` / pytest evaluation passes the full `Content` (text + `inline_data`) into `Runner.run_async`, and the same tool succeeds.

## Root cause

In `vertexai/_genai/_evals_common.py`:

### 1. DataFrame build flattens prompt to text

`_eval_cases_to_dataframe` stores:

```python
row[PROMPT] = _evals_data_converters._get_content_text(case.prompt)
```

`_get_content_text` (`_evals_data_converters.py`) concatenates `part.text` and **ignores** `inline_data`, `file_data`, etc.

### 2. Local agent run wraps string as single text Part

`_execute_local_agent_run_with_retry_async` calls:

```python
new_message_content = genai_types.Content(
role="user",
parts=[genai_types.Part(text=contents)],
)
```

So even if `contents` were a full `Content`, this stringifies it into one text part. Today `contents` is already the plain text string.

### 3. N+1 / `agent_data` path — same bug

When inference uses `agent_data` turns, the code still calls `_get_content_text(last_user_content)` before the local agent run.

## Expected behavior

Preserve full multimodal `Content` (all `parts`, including `inline_data` / `file_data`) through the local-agent inference path and call:

```python
Runner.run_async(new_message=Content.model_validate(full_content))
```

instead of `Part(text=...)`.

## Suggested fix locations

`vertexai/_genai/_evals_common.py`:

1. `_eval_cases_to_dataframe` — store full prompt `Content` dict (with `parts`) when multimodal, not `_get_content_text()` only.
2. `_extract_contents_for_inference` — if dict has `parts`, pass through (do not require nested `contents` key).
3. `_execute_local_agent_run_with_retry_async` — use `Content.model_validate(contents)` when `contents` is a Content dict; keep `Part(text=...)` fallback for plain strings.
4. N+1 `agent_data` branch — same helper instead of `_get_content_text(last_user_content)`.

## Impact

Blocks multimodal agent eval via `agents-cli eval generate` for any tool that reads uploaded bytes from `user_content` (PDF invoices, images, etc.). Workaround today is ADK `AgentEvaluator` pytest path, which preserves `inline_data`.

## References

- agents-cli multimodal eval schema: https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/references/multimodal-eval.md
- Related open eval issues: #6785

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.