Azure / Azure/azure-sdk-for-python
Code-based evaluators show `score` as `Error` in Azure Foundry evaluations
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: `azure-ai-projects`
- **Package Version**: 2.0.0
- **Operating System**: Ubuntu
- **Python Version**: 3.10
**Describe the bug**
There are a couple of issues with the `ai-projects-sdk` when running custom evaluators.
- When running the sample for a custom code-based evaluator [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py), the UI in Foundry creates two metrics instead of one. The `score` is shown as "Error".
- Logging output is not shown in the user logs, which makes it hard to debug custom scoring logic
**To Reproduce**
Steps to reproduce the behavior:
1. Run the [sample code](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py)
The `my_custom_evaluator_code: score` column shows an error:
**Note**: the current sample code in the repository uses somewhat questionable queries, but I did not modify those for the purpose of running the sample in an unmodified manner. It may be worth considering changing the queries in the sample to something more neutral.
**Expected behavior**
- There should be an explanation of why this creates two metrics
- The score should not display as "Error" when using the SDK
- The sample should use neutral questions
Contributor guide
Assessment
This issue has not been assessed yet.