Azure / Azure/azure-sdk-for-python

azure-ai-evaluation: Document Retrieval Evaluator not recognized despite being listed as built-in in AI Projects SDK

Open
#42,736 3 comments 0 reactions 1 assignee Claimed by @singankit View on GitHub
AI Projects Client customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

**Package Name:** azure-ai-projects
**Package Version:** azure-ai-projects==1.0.0b11
**Operating System:** Windows
**Python Version:** Python3.12
azure-ai-evaluation: Document Retrieval Evaluator not recognized despite being listed as built-in in AI Projects SDK

**Describe the bug**
This is a near-identical issue to [here](https://github.com/Azure/azure-sdk-for-python/issues/41869). Unable to run document retrieval evaluator using azure-ai-projects. The evaluator is present in the EvaluatorIds enum in the SDK, yet is not present in the library in Foundry.

**To Reproduce**
Steps to reproduce the behavior:
```
evaluator = EvaluatorConfiguration(
id=EvaluatorIds.DOCUMENT_RETRIEVAL.value,
init_params={"deployment_name": deployment_name},
)
```

**Expected behavior**
Document Retrieval Evaluator should be used to evaluate the data

**Additional context**
This is the stacktrace in foundry logs:

```
Traceback (most recent call last):
File "/app/evaluation/evaluation.py", line 28, in run_evaluation
results = execute_evaluators(evaluation, input_path, output_path, logger, log_path)
File "/app/evaluation/eval_utils.py", line 271, in execute_evaluators
evaluators_dict = initialize_evaluators(evaluation, evaluation.service_based_evaluators, logger)
File "/app/evaluation/eval_utils.py", line 204, in initialize_evaluators
evaluator_constructor = BuiltInEvaluatorConstructor(logger, evaluator_configuration, service_based_evaluators, project_url=properties.base_uri, azureai_token=properties.azureai_token)
File "/app/evaluation/eval_utils.py", line 100, in __init__
self.evaluator = self._initialize_evaluator()
File "/app/evaluation/eval_utils.py", line 156, in _initialize_evaluator
raise ValueError(f"Unknown evaluator id: {evaluator_name}")
ValueError: Unknown evaluator id: document_retrieval
```

Document Retrieval is a valid enum listed [here](https://github.com/Azure/azure-sdk-for-python/blob/0f401b0642308a3ddc834adf4303a08f2c1d9639/sdk/ai/azure-ai-projects/azure/ai/projects/models/_patch_evaluations.py#L38)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.