googleapis / googleapis/python-aiplatform

response_evaluation_score (COHERENCE) fails with INVALID_ARGUMENT: Error parsing JSON in all regions — vertexai 1.153.1

未关闭
#6,853 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

## Bug description

When using `PrebuiltMetric.COHERENCE` via `client.evals.evaluate()` (which backs `response_evaluation_score` in google-adk), the call consistently fails with:

```
400 INVALID_ARGUMENT: Error parsing JSON. Last error: Expecting property name
enclosed in double quotes: line 1 column 2 (char 1), Input: {* **STEP 1: ...}
```

The autorater LLM generates a free-text response starting with `{*` (bullet points), and the `:evaluateInstances` endpoint attempts to parse it as JSON and fails.

## Reproduction

```python
import vertexai
import pandas as pd

client = vertexai.Client(project="", location="us-central1")

df = pd.DataFrame([{
"prompt": "What slabs are in the model?",
"response": "The model has 17 C25 concrete slabs."
}])

result = client.evals.evaluate(
dataset=df,
metrics=[vertexai.types.PrebuiltMetric.COHERENCE],
)
```

## Environment

- `google-cloud-aiplatform` (vertexai): **1.153.1**
- Regions tested: `us-central1`, `europe-west1`, `europe-west3` — all fail
- The COHERENCE metric YAML (`gs://vertex-ai-generative-ai-eval-sdk-resources/metrics/coherence/v1.yaml`) produces free-text output, but the `:evaluateInstances` server endpoint attempts to parse the autorater response as JSON and fails

## Expected behavior

Metric evaluation completes and returns a coherence score.

## Actual behavior

`num_cases_error=1`, `num_cases_valid=0` on every call. The full error from the server:

```
google.genai.errors.ClientError: 400 INVALID_ARGUMENT.
Error parsing JSON. Last error: Expecting property name enclosed in double quotes:
line 1 column 2 (char 1), Input: {* **STEP 1: Identify the purpose and audience:...
```

The autorater generates a valid free-text evaluation but the server-side JSON parser rejects it.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。