Azure / Azure/azure-sdk-for-python

Code-based evaluators show `score` as `Error` in Azure Foundry evaluations

Open
#45,643 4 comments 0 reactions 1 assignee Assigned to @w-javed View on GitHub
AI Projects customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: `azure-ai-projects`
- **Package Version**: 2.0.0
- **Operating System**: Ubuntu
- **Python Version**: 3.10

**Describe the bug**

There are a couple of issues with the `ai-projects-sdk` when running custom evaluators.

- When running the sample for a custom code-based evaluator [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py), the UI in Foundry creates two metrics instead of one. The `score` is shown as "Error".
- Logging output is not shown in the user logs, which makes it hard to debug custom scoring logic

**To Reproduce**

Steps to reproduce the behavior:
1. Run the [sample code](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py)

The `my_custom_evaluator_code: score` column shows an error:

Image

**Note**: the current sample code in the repository uses somewhat questionable queries, but I did not modify those for the purpose of running the sample in an unmodified manner. It may be worth considering changing the queries in the sample to something more neutral.

**Expected behavior**

- There should be an explanation of why this creates two metrics
- The score should not display as "Error" when using the SDK
- The sample should use neutral questions

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.