googleapis / googleapis/python-aiplatform
Support Flexible Inputs to vertexai.types.Metric
- Lingua principale
- Python
- Stelle
- 905
- Fork
- 465
- Merge medio
- 1g 13h
- PR unite (30g)
- 44
Descrizione
Many evaluation metrics do not require extra context provided by prompts. In fact, the prompt may confuse the metric. For example, a non_advice metric might start checking response on instruction-following confusing instructions for extensions to non_advice criteria.
My use case for evaluating responses without relying on a prompt is currently not supported.
Hardcoding an LLMMetric with only a {response} field, and
- supplying an eval_dataset with only a response column raises EvalDatasetSchema.UNKNOWN
- while supplying an eval_dataset with prompt and response columns raises INVALID_ARGUMENT
The current workarounds (supplying dummy prompt or extra 'ignore prompt' instructions to the judge) are cumbersome.
I propose adding infrastructure to support the bare minimum 'response' input, in the form of a more flexible `evaluate` -- so addressing the INVALID_ARGUMENT error.
This would be a welcome addition to the existing infrastructure supporting extra inputs.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.