googleapis / googleapis/python-aiplatform
response_evaluation_score (COHERENCE) fails with INVALID_ARGUMENT: Error parsing JSON in all regions — vertexai 1.153.1
- Lingua principale
- Python
- Stelle
- 905
- Fork
- 465
- Merge medio
- 1g 13h
- PR unite (30g)
- 44
Descrizione
## Bug description
When using `PrebuiltMetric.COHERENCE` via `client.evals.evaluate()` (which backs `response_evaluation_score` in google-adk), the call consistently fails with:
```
400 INVALID_ARGUMENT: Error parsing JSON. Last error: Expecting property name
enclosed in double quotes: line 1 column 2 (char 1), Input: {* **STEP 1: ...}
```
The autorater LLM generates a free-text response starting with `{*` (bullet points), and the `:evaluateInstances` endpoint attempts to parse it as JSON and fails.
## Reproduction
```python
import vertexai
import pandas as pd
client = vertexai.Client(project="", location="us-central1")
df = pd.DataFrame([{
"prompt": "What slabs are in the model?",
"response": "The model has 17 C25 concrete slabs."
}])
result = client.evals.evaluate(
dataset=df,
metrics=[vertexai.types.PrebuiltMetric.COHERENCE],
)
```
## Environment
- `google-cloud-aiplatform` (vertexai): **1.153.1**
- Regions tested: `us-central1`, `europe-west1`, `europe-west3` — all fail
- The COHERENCE metric YAML (`gs://vertex-ai-generative-ai-eval-sdk-resources/metrics/coherence/v1.yaml`) produces free-text output, but the `:evaluateInstances` server endpoint attempts to parse the autorater response as JSON and fails
## Expected behavior
Metric evaluation completes and returns a coherence score.
## Actual behavior
`num_cases_error=1`, `num_cases_valid=0` on every call. The full error from the server:
```
google.genai.errors.ClientError: 400 INVALID_ARGUMENT.
Error parsing JSON. Last error: Expecting property name enclosed in double quotes:
line 1 column 2 (char 1), Input: {* **STEP 1: Identify the purpose and audience:...
```
The autorater generates a valid free-text evaluation but the server-side JSON parser rejects it.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.