googleapis / googleapis/python-aiplatform

[bug] client.evals.create_evaluation_run(agent=<reasoningEngine>) fails for A2A engines — hardcodes stream_query() on agent_engine that only exposes on_message_send

Offen
#6,837 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @maxgasztych Auf GitHub ansehen
api: vertex-ai
Vorherrschende Sprache
Python
Sterne
905
Forks
465
Ø Merge
1 T. 13 Std.
Gemergte PRs (30 T.)
44

Beschreibung

## TL;DR

`vertexai._genai._evals_common._execute_agent_run_with_retry` unconditionally calls `agent_engine.stream_query(...)`, but Agent Engines deployed with the [A2A template](https://google.github.io/adk-docs/a2a/) (`vertexai.preview.reasoning_engines.A2aAgent` / `agents-cli --agent adk_a2a`) expose only `on_message_send` (`api_mode=a2a_extension`). The resulting `AttributeError` is swallowed by a broad `except Exception`, surfaces as the opaque `INTERNAL: Internal error occurred.`, and the eval items never reach the engine container.

Repros 100% against any A2A engine; affects all `client.evals.create_evaluation_run(agent=...)` and `client.evals.run_inference(agent=...)` calls.

**Two distinct defects** in this report (see Root Cause):
1. Wrong method dispatch — A2A engines have no `stream_query`
2. Overbroad exception masking — the real `AttributeError` is invisible to the caller

A2A is a first-class Vertex Agent Engine deployment path ([docs](https://google.github.io/adk-docs/a2a/), [SDK
reference](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/develop/adk-a2a)), not an edge case — so failing silently here breaks a documented happy path.

## Versions

- `google-cloud-aiplatform==1.148.1` (also present at HEAD of the affected files in 1.149.x – 1.153.x)
- `a2a-sdk==0.3.26`
- Python 3.12, Linux + macOS

## Reproduce

```python
from vertexai import Client
from vertexai._genai import types as evals_types
import pandas as pd

client = Client(project="", location="us-central1")
run = client.evals.create_evaluation_run(
dataset=evals_types.EvaluationDataset(
eval_dataset_df=pd.DataFrame({"prompt": ["hello"]})
),
dest="gs:///eval/",
metrics=[evals_types.RubricMetric.GENERAL_QUALITY],
agent="projects//locations/us-central1/reasoningEngines/",
)
# Run state: FAILED
# Item errors: "INTERNAL: Internal error occurred."
# Engine container: zero log entries during the inference window
```

The swallowed exception under the opaque `INTERNAL` (captured by calling `stream_query` directly against the engine):

```
Traceback (most recent call last):
File ".../vertexai/_genai/_evals_common.py", line 2058, in _execute_agent_run_with_retry
for event in agent_engine.stream_query(
^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'AgentEngine' object has no attribute 'stream_query'
```

This bubbles through the retry loop and is returned as `{"error": "Failed after retries: ..."}`, then surfaced to the user as `INTERNAL: Internal
error occurred.` with no `AttributeError` mention.

## Root cause

### Defect 1: Wrong method dispatch (`_evals_common.py:2058`)

```python
for event in agent_engine.stream_query( # type: ignore[attr-defined]
user_id=user_id,
session_id=session_id,
message=contents,
):
```

There's no branching on the engine's API mode. A2A engines' `operation_schemas()` reports:

```json
[
{
"name": "on_message_send",
"api_mode": "a2a_extension",
"parameters": {"type": "object", "required": ["request", "context"]}
}
]
```

Verified live:
```python
ae = client.agent_engines.get(name=A2A_ENGINE_RESOURCE)
hasattr(ae, "stream_query") # False
hasattr(ae, "on_message_send") # True
```

### Defect 2: Overbroad exception masking (`_evals_common.py:2078-2089`)

```python
except Exception as e: # pylint: disable=broad-exception-caught
logger.error("Unexpected error during generate_content on attempt %d/%d: %s", ...)
if attempt == max_retries - 1:
return {"error": f"Failed after retries: {e}"}
```

The bare `except Exception` plus 3 retries means a deterministic AttributeError surfaces as a generic post-retry error string. The original exception type is lost from the public error path; a user has to read SDK source to figure out what's happening.

## Two acceptable outcomes for the fix

I'd consider either of these resolving the report:

### Option A — Add A2A dispatch path

In `_execute_agent_run_with_retry`, detect A2A via `operation_schemas()` and dispatch via `on_message_send` (Vertex A2A proto-JSON
`/a2a/v1/message:send`):

```python
schemas = agent_engine.operation_schemas()
a2a_method = next(
(s["name"] for s in schemas if s.get("api_mode") == "a2a_extension"),
None,
)
if a2a_method:
# Dispatch via A2A message:send and translate the response to the
# eval service's expected event shape.
...
else:
for event in agent_engine.stream_query(...):
...
```

### Option B — Explicit unsupported error

If A2A inference is out of scope for the eval SDK in the near term, fail fast at `create_evaluation_run` / `run_inference` time with an actionable
error:

```python
schemas = agent_engine.operation_schemas()
if any(s.get("api_mode") == "a2a_extension" for s in schemas):
raise NotImplementedError(
"A2A agent engines are not yet supported by client.evals inference. "
"Use the two-step workaround: invoke the engine directly via "
"with pre-populated 'response' column."
)
```

In both options, the broad `except Exception` at line 2078 should be narrowed (or moved to catch only transport / quota retryable errors) so genuine programming defects don't surface as `INTERNAL`.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.