Azure / Azure/azure-sdk-for-python
azure-ai-evaluation: Document Retrieval Evaluator not recognized despite being listed as built-in in AI Projects SDK
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
**Package Name:** azure-ai-projects
**Package Version:** azure-ai-projects==1.0.0b11
**Operating System:** Windows
**Python Version:** Python3.12
azure-ai-evaluation: Document Retrieval Evaluator not recognized despite being listed as built-in in AI Projects SDK
**Describe the bug**
This is a near-identical issue to [here](https://github.com/Azure/azure-sdk-for-python/issues/41869). Unable to run document retrieval evaluator using azure-ai-projects. The evaluator is present in the EvaluatorIds enum in the SDK, yet is not present in the library in Foundry.
**To Reproduce**
Steps to reproduce the behavior:
```
evaluator = EvaluatorConfiguration(
id=EvaluatorIds.DOCUMENT_RETRIEVAL.value,
init_params={"deployment_name": deployment_name},
)
```
**Expected behavior**
Document Retrieval Evaluator should be used to evaluate the data
**Additional context**
This is the stacktrace in foundry logs:
```
Traceback (most recent call last):
File "/app/evaluation/evaluation.py", line 28, in run_evaluation
results = execute_evaluators(evaluation, input_path, output_path, logger, log_path)
File "/app/evaluation/eval_utils.py", line 271, in execute_evaluators
evaluators_dict = initialize_evaluators(evaluation, evaluation.service_based_evaluators, logger)
File "/app/evaluation/eval_utils.py", line 204, in initialize_evaluators
evaluator_constructor = BuiltInEvaluatorConstructor(logger, evaluator_configuration, service_based_evaluators, project_url=properties.base_uri, azureai_token=properties.azureai_token)
File "/app/evaluation/eval_utils.py", line 100, in __init__
self.evaluator = self._initialize_evaluator()
File "/app/evaluation/eval_utils.py", line 156, in _initialize_evaluator
raise ValueError(f"Unknown evaluator id: {evaluator_name}")
ValueError: Unknown evaluator id: document_retrieval
```
Document Retrieval is a valid enum listed [here](https://github.com/Azure/azure-sdk-for-python/blob/0f401b0642308a3ddc834adf4303a08f2c1d9639/sdk/ai/azure-ai-projects/azure/ai/projects/models/_patch_evaluations.py#L38)
Contributor guide
Assessment
This issue has not been assessed yet.