googleapis / googleapis/python-aiplatform

response_evaluation_score (COHERENCE) fails with INVALID_ARGUMENT: Error parsing JSON in all regions — vertexai 1.153.1

Abierto
#6,853 0 comentarios 0 reacciones 0 asignados Ver en GitHub
api: vertex-ai
Lenguaje dominante
Python
Estrellas
905
Forks
465
Merge medio
1 d 13 h
PR fusionados (30 d)
44

Descripción

## Bug description

When using `PrebuiltMetric.COHERENCE` via `client.evals.evaluate()` (which backs `response_evaluation_score` in google-adk), the call consistently fails with:

```
400 INVALID_ARGUMENT: Error parsing JSON. Last error: Expecting property name
enclosed in double quotes: line 1 column 2 (char 1), Input: {* **STEP 1: ...}
```

The autorater LLM generates a free-text response starting with `{*` (bullet points), and the `:evaluateInstances` endpoint attempts to parse it as JSON and fails.

## Reproduction

```python
import vertexai
import pandas as pd

client = vertexai.Client(project="", location="us-central1")

df = pd.DataFrame([{
"prompt": "What slabs are in the model?",
"response": "The model has 17 C25 concrete slabs."
}])

result = client.evals.evaluate(
dataset=df,
metrics=[vertexai.types.PrebuiltMetric.COHERENCE],
)
```

## Environment

- `google-cloud-aiplatform` (vertexai): **1.153.1**
- Regions tested: `us-central1`, `europe-west1`, `europe-west3` — all fail
- The COHERENCE metric YAML (`gs://vertex-ai-generative-ai-eval-sdk-resources/metrics/coherence/v1.yaml`) produces free-text output, but the `:evaluateInstances` server endpoint attempts to parse the autorater response as JSON and fails

## Expected behavior

Metric evaluation completes and returns a coherence score.

## Actual behavior

`num_cases_error=1`, `num_cases_valid=0` on every call. The full error from the server:

```
google.genai.errors.ClientError: 400 INVALID_ARGUMENT.
Error parsing JSON. Last error: Expecting property name enclosed in double quotes:
line 1 column 2 (char 1), Input: {* **STEP 1: Identify the purpose and audience:...
```

The autorater generates a valid free-text evaluation but the server-side JSON parser rejects it.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.