Azure / Azure/azure-sdk-for-python

Custom Evaluators Can’t Run in Cloud: Docs Require Hub, Latest SDK Drops Hub Support

Abierto
#42,201 2 comentarios 0 reacciones 1 asignado Reclamado por @singankit Ver en GitHub
AI AI Projects customer-reported Evaluation feature-request needs-team-attention Service Attention
Lenguaje dominante
Python
Estrellas
5.6k
Forks
3.4k
Merge medio
2 d 2 h
PR fusionados (30 d)
213

Descripción

- **Package Name**: azure-ai-evaluation, azure-ai-projects
- **Package Version**: 1.9.0, 1.0.0b12
- **Operating System**: Windows
- **Python Version**: 3.13.3

**Describe the bug**
The official “Run evaluations in the cloud” guide () says custom evaluators *require* an **Azure AI Foundry hub project** (see the “Azure AI Foundry projects aren't supported” note).
Yet the current SDK release note on PyPI () states:

> “Support for … **hub‑based projects has been discontinued**. We recommend creating a new Azure AI Foundry resource utilizing a project endpoint. If this is not possible, pin azure‑ai‑projects to 1.0.0b10 or earlier.”

Because the SDK no longer recognizes hubs while the docs demand one, there is **no supported path to run remote evaluations with custom evaluators**.

Local evaluation is an unreliable fallback on Windows due to OSError #41913, which forces every evaluator into a separate run.

Relevant documentation links showing the conflict

| Link | Key statement |
|---|---|
| Cloud evaluation guide | “Azure AI Foundry projects aren't supported … use a hub project instead.” |
| Hub overview | Explains that hubs group projects and are needed for certain features. |
| SDK overview | “This article applies to a Foundry project. The code shown here doesn't work for a hub based project.” |
| PyPI 1.0.0b12 note | Hub support removed. |
| PyPI 1.0.0b10 | Last release that still supports hubs. |
| Local evaluation guide | Recommends local runs when remote not possible. |
| Agent‑evaluation guide | Advises migrating off hub‑only SDKs. |
| azure‑ai‑evaluation PyPI | Still promotes “built‑in **or custom** evaluators”. |
| GitHub issue #41913 | Demonstrates Windows‑specific OSError that makes local evaluation unwieldy. |

**Expected behavior**
Either
* the SDK should still connect to hub projects so custom‑evaluator cloud runs keep working, **or**
* the documentation must be updated and a new, hub‑free mechanism provided for registering & invoking custom evaluators on plain Foundry projects.

**Screenshots**
N/A – this is a doc/SDK incompatibility.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.