googleapis / googleapis/python-aiplatform

Support Flexible Inputs to vertexai.types.Metric

Ouverte
#6,261 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
api: vertex-ai
Langage dominant
Python
Étoiles
905
Forks
465
Merge moyen
1 j 13 h
PR mergées (30 j)
44

Description

Many evaluation metrics do not require extra context provided by prompts. In fact, the prompt may confuse the metric. For example, a non_advice metric might start checking response on instruction-following confusing instructions for extensions to non_advice criteria.

My use case for evaluating responses without relying on a prompt is currently not supported.

Hardcoding an LLMMetric with only a {response} field, and

- supplying an eval_dataset with only a response column raises EvalDatasetSchema.UNKNOWN
- while supplying an eval_dataset with prompt and response columns raises INVALID_ARGUMENT

The current workarounds (supplying dummy prompt or extra 'ignore prompt' instructions to the judge) are cumbersome.

I propose adding infrastructure to support the bare minimum 'response' input, in the form of a more flexible `evaluate` -- so addressing the INVALID_ARGUMENT error.

This would be a welcome addition to the existing infrastructure supporting extra inputs.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.