Azure / Azure/azure-sdk-for-python

Code-based evaluators show `score` as `Error` in Azure Foundry evaluations

Abierto
#45,643 4 comentarios 0 reacciones 1 asignado Asignado a @w-javed Ver en GitHub
AI Projects customer-reported needs-team-attention question Service Attention
Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
2 d
PR fusionados (30 d)
217

Descripción

- **Package Name**: `azure-ai-projects`
- **Package Version**: 2.0.0
- **Operating System**: Ubuntu
- **Python Version**: 3.10

**Describe the bug**

There are a couple of issues with the `ai-projects-sdk` when running custom evaluators.

- When running the sample for a custom code-based evaluator [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py), the UI in Foundry creates two metrics instead of one. The `score` is shown as "Error".
- Logging output is not shown in the user logs, which makes it hard to debug custom scoring logic

**To Reproduce**

Steps to reproduce the behavior:
1. Run the [sample code](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py)

The `my_custom_evaluator_code: score` column shows an error:

Image

**Note**: the current sample code in the repository uses somewhat questionable queries, but I did not modify those for the purpose of running the sample in an unmodified manner. It may be worth considering changing the queries in the sample to something more neutral.

**Expected behavior**

- There should be an explanation of why this creates two metrics
- The score should not display as "Error" when using the SDK
- The sample should use neutral questions

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.