Azure / Azure/azure-sdk-for-python
Code-based evaluators show `score` as `Error` in Azure Foundry evaluations
- Lingua principale
- Python
- Stelle
- 5.6k
- Fork
- 3.4k
- Merge medio
- 2g 2h
- PR unite (30g)
- 202
Descrizione
- **Package Name**: `azure-ai-projects`
- **Package Version**: 2.0.0
- **Operating System**: Ubuntu
- **Python Version**: 3.10
**Describe the bug**
There are a couple of issues with the `ai-projects-sdk` when running custom evaluators.
- When running the sample for a custom code-based evaluator [here](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py), the UI in Foundry creates two metrics instead of one. The `score` is shown as "Error".
- Logging output is not shown in the user logs, which makes it hard to debug custom scoring logic
**To Reproduce**
Steps to reproduce the behavior:
1. Run the [sample code](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_code_based_evaluators.py)
The `my_custom_evaluator_code: score` column shows an error:
**Note**: the current sample code in the repository uses somewhat questionable queries, but I did not modify those for the purpose of running the sample in an unmodified manner. It may be worth considering changing the queries in the sample to something more neutral.
**Expected behavior**
- There should be an explanation of why this creates two metrics
- The score should not display as "Error" when using the SDK
- The sample should use neutral questions
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.