Azure / Azure/azure-sdk-for-python

Custom Evaluators Can’t Run in Cloud: Docs Require Hub, Latest SDK Drops Hub Support

Open
#42,201 2 comments 0 reactions 1 assignee Claimed by @singankit View on GitHub
AI AI Projects customer-reported Evaluation feature-request needs-team-attention Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

- **Package Name**: azure-ai-evaluation, azure-ai-projects
- **Package Version**: 1.9.0, 1.0.0b12
- **Operating System**: Windows
- **Python Version**: 3.13.3

**Describe the bug**
The official “Run evaluations in the cloud” guide () says custom evaluators *require* an **Azure AI Foundry hub project** (see the “Azure AI Foundry projects aren't supported” note).
Yet the current SDK release note on PyPI () states:

> “Support for … **hub‑based projects has been discontinued**. We recommend creating a new Azure AI Foundry resource utilizing a project endpoint. If this is not possible, pin azure‑ai‑projects to 1.0.0b10 or earlier.”

Because the SDK no longer recognizes hubs while the docs demand one, there is **no supported path to run remote evaluations with custom evaluators**.

Local evaluation is an unreliable fallback on Windows due to OSError #41913, which forces every evaluator into a separate run.

Relevant documentation links showing the conflict

| Link | Key statement |
|---|---|
| Cloud evaluation guide | “Azure AI Foundry projects aren't supported … use a hub project instead.” |
| Hub overview | Explains that hubs group projects and are needed for certain features. |
| SDK overview | “This article applies to a Foundry project. The code shown here doesn't work for a hub based project.” |
| PyPI 1.0.0b12 note | Hub support removed. |
| PyPI 1.0.0b10 | Last release that still supports hubs. |
| Local evaluation guide | Recommends local runs when remote not possible. |
| Agent‑evaluation guide | Advises migrating off hub‑only SDKs. |
| azure‑ai‑evaluation PyPI | Still promotes “built‑in **or custom** evaluators”. |
| GitHub issue #41913 | Demonstrates Windows‑specific OSError that makes local evaluation unwieldy. |

**Expected behavior**
Either
* the SDK should still connect to hub projects so custom‑evaluator cloud runs keep working, **or**
* the documentation must be updated and a new, hub‑free mechanism provided for registering & invoking custom evaluators on plain Foundry projects.

**Screenshots**
N/A – this is a doc/SDK incompatibility.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.