Azure / Azure/azure-sdk-for-python
Custom Evaluators Can’t Run in Cloud: Docs Require Hub, Latest SDK Drops Hub Support
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
- **Package Name**: azure-ai-evaluation, azure-ai-projects
- **Package Version**: 1.9.0, 1.0.0b12
- **Operating System**: Windows
- **Python Version**: 3.13.3
**Describe the bug**
The official “Run evaluations in the cloud” guide () says custom evaluators *require* an **Azure AI Foundry hub project** (see the “Azure AI Foundry projects aren't supported” note).
Yet the current SDK release note on PyPI () states:
> “Support for … **hub‑based projects has been discontinued**. We recommend creating a new Azure AI Foundry resource utilizing a project endpoint. If this is not possible, pin azure‑ai‑projects to 1.0.0b10 or earlier.”
Because the SDK no longer recognizes hubs while the docs demand one, there is **no supported path to run remote evaluations with custom evaluators**.
Local evaluation is an unreliable fallback on Windows due to OSError #41913, which forces every evaluator into a separate run.
Relevant documentation links showing the conflict
| Link | Key statement |
|---|---|
| Cloud evaluation guide | “Azure AI Foundry projects aren't supported … use a hub project instead.” |
| Hub overview | Explains that hubs group projects and are needed for certain features. |
| SDK overview | “This article applies to a Foundry project. The code shown here doesn't work for a hub based project.” |
| PyPI 1.0.0b12 note | Hub support removed. |
| PyPI 1.0.0b10 | Last release that still supports hubs. |
| Local evaluation guide | Recommends local runs when remote not possible. |
| Agent‑evaluation guide | Advises migrating off hub‑only SDKs. |
| azure‑ai‑evaluation PyPI | Still promotes “built‑in **or custom** evaluators”. |
| GitHub issue #41913 | Demonstrates Windows‑specific OSError that makes local evaluation unwieldy. |
**Expected behavior**
Either
* the SDK should still connect to hub projects so custom‑evaluator cloud runs keep working, **or**
* the documentation must be updated and a new, hub‑free mechanism provided for registering & invoking custom evaluators on plain Foundry projects.
**Screenshots**
N/A – this is a doc/SDK incompatibility.
Contributor guide
Assessment
This issue has not been assessed yet.