googleapis / googleapis/python-aiplatform

Shipped: vertexai-openeval-adapter — EvalPort import/export for vertexai.evaluation results

オープン
#7,078 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
api: vertex-ai
主要言語
Python
スター
905
フォーク
465
平均マージ
1日 13時間
マージ済み PR(30日)
44

説明

Built and shipped a standalone adapter that converts `vertexai.evaluation` metrics and `EvalResult`s to and from [EvalPort](https://github.com/adhabnr-ux/evalport) (Apache 2.0) — an open interchange format for portable LLM evaluation datasets (test cases, graders, suites, and result sets as plain JSON). It's already integrated with UK AISI's Inspect AI (PR merged) and has standalone adapter packages for a dozen+ eval/observability frameworks (Ragas, LangSmith, MLflow, Braintrust, DeepEval-adjacent tools aside — AutoGen, CrewAI, Langfuse, Evidently, TruLens, Opik, Giskard, Argilla), so a Vertex AI Gen AI Evaluation Service adapter puts it in company with the rest of that ecosystem.

**[`vertexai-openeval-adapter`](https://github.com/adhabnr-ux/evalport/tree/main/adapters/vertexai-openeval-adapter)**

```python
import pandas as pd
from vertexai.evaluation import EvalTask, PointwiseMetric, PointwiseMetricPromptTemplate
from vertexai_openeval_adapter import to_openeval, from_openeval, eval_result_to_openeval

dataset = pd.DataFrame({"prompt": ["What is the capital of France?"], "reference": ["Paris"]})
suite = to_openeval(dataset, input_column="prompt", expected_output_column="reference", suite_id="geo_quiz")

from openeval.validate import validate_suite
assert validate_suite(suite).valid

quality_metric = PointwiseMetric(
metric="quality",
metric_prompt_template=PointwiseMetricPromptTemplate(
criteria="Is the response factually correct?", metric_definition="Factual accuracy"
),
)
eval_task = EvalTask(dataset=dataset, metrics=[quality_metric])
result = eval_task.evaluate()

result_set = eval_result_to_openeval(result, suite_id="geo_quiz", run_id="run-1", started_at="2026-08-16T00:00:00Z")
assert validate_result_set(result_set).valid
```

The metric-mapping is the part I'd flag as genuinely interesting rather than routine: `PointwiseMetric` maps to EvalPort's `llm_judge` grader with the **actual rendered prompt template** preserved verbatim in the grader's `params.prompt_template` (read directly from `PointwiseMetricPromptTemplate`'s own rendering, not reconstructed or guessed) — so a suite exported from Vertex AI carries the real judge instructions, not a placeholder. `CustomMetric` and `PairwiseMetric` are exported as `custom`-typed graders (execution-only, not reconstructed on import) since both compute client-side per Vertex's own docstrings and have no portable representation. Raw string metric names (`"rouge_1"`, `"bleu"`, etc.) are explicitly rejected with a `TypeError` rather than silently guessed at, since their scoring logic isn't introspectable from the SDK's own objects. The adapter reads `EvalResult.metrics_table` using Vertex's own column convention (`f"{metric_name}/score"`), verified directly against `vertexai/evaluation/_evaluation.py` source rather than assumed.

Tested against the real `google-cloud-aiplatform[evaluation]` package (not mocks) and the real `openeval.validate.validate_suite()`/`validate_result_set()`. Full README with the complete mapping table and round-trip notes: https://github.com/adhabnr-ux/evalport/tree/main/adapters/vertexai-openeval-adapter#readme

No action needed here — this lives entirely as an external package (`pip install vertexai-openeval-adapter`), zero footprint on `google-cloud-aiplatform` itself. Flagging mainly so it's discoverable; happy to adjust the mapping if the evaluation module's public API shifts.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

This issue describes the external vertexai-openeval-adapter package rather than a change to python-aiplatform. Start with the linked adapter README and package path to understand its Vertex AI and EvalPort mappings. There is no in-repository entry point, test target, or acceptance criterion for a contributor to complete.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
machine-learning
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
静か
明瞭さ
説明が足りない
初心者へのやさしさ
10/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。