microsoft / microsoft/agent-framework

.NET: Bug: FoundryEvals sends `azure_ai_evaluator` testing-criteria type — HTTP 400 from current Foundry endpoints

Open
#6,991 3 comments 0 reactions 1 assignee View on GitHub

@alliscode is already working on this.

Since Jul 8, 2026.

.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

## Summary

`FoundryEvals.EvaluateAsync` always returns **HTTP 400** against current Azure AI Foundry project endpoints (`*.services.ai.azure.com/api/projects/*`). The service rejects the request body with:

```
HTTP 400 (invalid_request_error)
Invalid set of testing criteria. Invalid value: 'azure_ai_evaluator'.
Supported values are: [label_model, text_similarity, string_check, label_model, score_model, python, endpoint]
```

## Root Cause

In `dotnet/src/Microsoft.Agents.AI.Foundry/Evaluation/FoundryEvalWireModels.cs`, the `WireTestingCriterion` type defaults to `"azure_ai_evaluator"`:

```csharp
internal sealed class WireTestingCriterion
{
[JsonPropertyName("type")]
public string Type { get; init; } = "azure_ai_evaluator"; // ← rejected by current API
...
}
```

The current Foundry Evals REST API at `*.services.ai.azure.com` no longer accepts `azure_ai_evaluator` as a criterion type. The bug is present in the current `main` branch and in the published nightly `1.12.0-preview.260629.1`.

## Repro

```csharp
// Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1
// Azure.AI.Projects 2.1.0-beta.3
// Endpoint: https://.services.ai.azure.com/api/projects/

AIProjectClient client = new(new Uri(endpoint), new DefaultAzureCredential());
FoundryEvals evals = new(client, "gpt-4o", FoundryEvals.Relevance, FoundryEvals.TaskAdherence);

// Throws ClientResultException: HTTP 400 (invalid_request_error)
await agent.EvaluateAsync(queries, evals);
```

The Azure AD authentication succeeds. The failure is in the POST to `CreateEvaluationAsync` where the request body contains `"type": "azure_ai_evaluator"` in `testing_criteria`.

## Suggested Fix

Change the default or routing logic so built-in named evaluators use the supported criteria type (likely `"score_model"` with appropriate prompt configuration, or a new `azure_ai_evaluator`-equivalent type once the API exposes it). All three evaluation paths (`EvaluateAsync`, `EvaluateTracesAsync`, `EvaluateFoundryTargetAsync`) call `FoundryEvalConverter.BuildTestingCriteria` which produces `WireTestingCriterion` instances — fixing the wire model type fixes all paths.

## Environment

| Item | Value |
|---|---|
| Package | `Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1` |
| `Azure.AI.Projects` | `2.1.0-beta.3` |
| Foundry endpoint format | `https://.services.ai.azure.com/api/projects/` |
| .NET | `net10.0` |
| Auth | `DefaultAzureCredential` (working, HTTP 400 is not an auth error) |

## Workaround

Wrap `FoundryEvals` in a try/catch and degrade gracefully to local AgentEval metrics. See [AgentEval samples 12 & 13](https://github.com/AgentEvalHQ/AgentEval) for a reference implementation of this pattern.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.