Azure / Azure/azure-sdk-for-python

Code-based evaluators show `score` as `Error` in Azure Foundry evaluations

Đang mở
#45,643 4 bình luận 0 reaction 1 người được giao Được giao cho @w-javed Xem trên GitHub
AI Projects customer-reported needs-team-attention question Service Attention
Ngôn ngữ chính
Python
Star
5.6k
Fork
3.4k
Merge trung bình
2 ngày 2 giờ
Pull request đã merge (30 ngày)
213

Mô tả

- **Package Name**: `azure-ai-projects`
- **Package Version**: 2.0.0
- **Operating System**: Ubuntu
- **Python Version**: 3.10

**Describe the bug**

There are a couple of issues with the `ai-projects-sdk` when running custom evaluators.

- When running the sample for a custom code-based evaluator [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py), the UI in Foundry creates two metrics instead of one. The `score` is shown as "Error".
- Logging output is not shown in the user logs, which makes it hard to debug custom scoring logic

**To Reproduce**

Steps to reproduce the behavior:
1. Run the [sample code](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py)

The `my_custom_evaluator_code: score` column shows an error:

Image

**Note**: the current sample code in the repository uses somewhat questionable queries, but I did not modify those for the purpose of running the sample in an unmodified manner. It may be worth considering changing the queries in the sample to something more neutral.

**Expected behavior**

- There should be an explanation of why this creates two metrics
- The score should not display as "Error" when using the SDK
- The sample should use neutral questions

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.