googleapis / googleapis/python-aiplatform

Support Flexible Inputs to vertexai.types.Metric

Open
#6,261 0 comments 0 reactions 0 assignees View on GitHub
api: vertex-ai
Dominant language
Python
Stars
905
Forks
465
Avg merge
1d 13h
Merged PRs (30d)
44

Description

Many evaluation metrics do not require extra context provided by prompts. In fact, the prompt may confuse the metric. For example, a non_advice metric might start checking response on instruction-following confusing instructions for extensions to non_advice criteria.

My use case for evaluating responses without relying on a prompt is currently not supported.

Hardcoding an LLMMetric with only a {response} field, and

- supplying an eval_dataset with only a response column raises EvalDatasetSchema.UNKNOWN
- while supplying an eval_dataset with prompt and response columns raises INVALID_ARGUMENT

The current workarounds (supplying dummy prompt or extra 'ignore prompt' instructions to the judge) are cumbersome.

I propose adding infrastructure to support the bare minimum 'response' input, in the form of a more flexible `evaluate` -- so addressing the INVALID_ARGUMENT error.

This would be a welcome addition to the existing infrastructure supporting extra inputs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.