googleapis / googleapis/python-aiplatform

Support Flexible Inputs to vertexai.types.Metric

未关闭
#6,261 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

Many evaluation metrics do not require extra context provided by prompts. In fact, the prompt may confuse the metric. For example, a non_advice metric might start checking response on instruction-following confusing instructions for extensions to non_advice criteria.

My use case for evaluating responses without relying on a prompt is currently not supported.

Hardcoding an LLMMetric with only a {response} field, and

- supplying an eval_dataset with only a response column raises EvalDatasetSchema.UNKNOWN
- while supplying an eval_dataset with prompt and response columns raises INVALID_ARGUMENT

The current workarounds (supplying dummy prompt or extra 'ignore prompt' instructions to the judge) are cumbersome.

I propose adding infrastructure to support the bare minimum 'response' input, in the form of a more flexible `evaluate` -- so addressing the INVALID_ARGUMENT error.

This would be a welcome addition to the existing infrastructure supporting extra inputs.

贡献指南

打开贡献指南

调研方向

Start by tracing vertexai.types.Metric and the evaluate path described in the issue, then inspect how EvalDatasetSchema handles response-only and prompt-plus-response columns. Reproduce the UNKNOWN and INVALID_ARGUMENT cases with the two dataset shapes. Done means a response-only metric can be evaluated without requiring a prompt, while existing extra-input behavior remains supported.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
cloud, machine-learning
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。