Azure / Azure/azure-sdk-for-python

Support Custom Output Fields in Prompt-Based Custom Evaluators

Aperta
#44,368 2 commenti 0 reazioni 1 assegnatario Assegnata a @w-javed Vedi su GitHub
customer-reported Evaluation feature-request needs-team-attention question Service Attention
Lingua principale
Python
Stelle
5.6k
Fork
3.4k
Merge medio
2g 2h
PR unite (30g)
213

Descrizione

**Is your feature request related to a problem? Please describe.**

Custom evaluators in Azure AI Foundry currently support only two output fields: result and reason. This limits the evaluator’s ability to return richer, structured information that can help explain how a score was derived or provide actionable insights.
For example, when evaluating two generated test plans, I want the evaluator to surface details such as:

A list of missing test cases
A list of extra or unexpected test cases
Explanations for specific mismatches

Because the evaluator is prompt-based, it should ideally allow generating multiple output fields beyond just result and reason.
https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/custom-evaluators?view=foundry&preserve-view=true#example

**Describe the solution you'd like**
I would like support for custom, schema-defined output fields in prompt-based custom evaluators.
This would allow me to define additional structured outputs e.g., missing_plans, extra_plans,etc. and retrieve them directly in the evaluation result object.
The evaluator should allow specifying the expected output schema (similar to the example in the documentation) and return all fields in the evaluation response.

**Describe alternatives you've considered**
- Embedding all additional information inside the reason field, but this becomes messy, requires post-processing, and loses structure.

- Writing a fully custom evaluator agent, but that defeats the purpose of using prompt-based evaluators

**Additional context**
Add any other context or screenshots about the feature request here.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.