googleapis / googleapis/python-aiplatform
Support Flexible Inputs to vertexai.types.Metric
- 主要语言
- Python
- 星标
- 905
- 派生
- 465
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 44
描述
Many evaluation metrics do not require extra context provided by prompts. In fact, the prompt may confuse the metric. For example, a non_advice metric might start checking response on instruction-following confusing instructions for extensions to non_advice criteria.
My use case for evaluating responses without relying on a prompt is currently not supported.
Hardcoding an LLMMetric with only a {response} field, and
- supplying an eval_dataset with only a response column raises EvalDatasetSchema.UNKNOWN
- while supplying an eval_dataset with prompt and response columns raises INVALID_ARGUMENT
The current workarounds (supplying dummy prompt or extra 'ignore prompt' instructions to the judge) are cumbersome.
I propose adding infrastructure to support the bare minimum 'response' input, in the form of a more flexible `evaluate` -- so addressing the INVALID_ARGUMENT error.
This would be a welcome addition to the existing infrastructure supporting extra inputs.
贡献指南
调研方向
Start by tracing vertexai.types.Metric and the evaluate path described in the issue, then inspect how EvalDatasetSchema handles response-only and prompt-plus-response columns. Reproduce the UNKNOWN and INVALID_ARGUMENT cases with the two dataset shapes. Done means a response-only metric can be evaluated without requiring a prompt, while existing extra-input behavior remains supported.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- cloud, machine-learning
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 38/100