microsoft / microsoft/agent-framework
.NET: Bug: FoundryEvals sends `azure_ai_evaluator` testing-criteria type — HTTP 400 from current Foundry endpoints
@alliscode is already working on this.
Since Jul 8, 2026.
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
## Summary
`FoundryEvals.EvaluateAsync` always returns **HTTP 400** against current Azure AI Foundry project endpoints (`*.services.ai.azure.com/api/projects/*`). The service rejects the request body with:
```
HTTP 400 (invalid_request_error)
Invalid set of testing criteria. Invalid value: 'azure_ai_evaluator'.
Supported values are: [label_model, text_similarity, string_check, label_model, score_model, python, endpoint]
```
## Root Cause
In `dotnet/src/Microsoft.Agents.AI.Foundry/Evaluation/FoundryEvalWireModels.cs`, the `WireTestingCriterion` type defaults to `"azure_ai_evaluator"`:
```csharp
internal sealed class WireTestingCriterion
{
[JsonPropertyName("type")]
public string Type { get; init; } = "azure_ai_evaluator"; // ← rejected by current API
...
}
```
The current Foundry Evals REST API at `*.services.ai.azure.com` no longer accepts `azure_ai_evaluator` as a criterion type. The bug is present in the current `main` branch and in the published nightly `1.12.0-preview.260629.1`.
## Repro
```csharp
// Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1
// Azure.AI.Projects 2.1.0-beta.3
// Endpoint: https://.services.ai.azure.com/api/projects/
AIProjectClient client = new(new Uri(endpoint), new DefaultAzureCredential());
FoundryEvals evals = new(client, "gpt-4o", FoundryEvals.Relevance, FoundryEvals.TaskAdherence);
// Throws ClientResultException: HTTP 400 (invalid_request_error)
await agent.EvaluateAsync(queries, evals);
```
The Azure AD authentication succeeds. The failure is in the POST to `CreateEvaluationAsync` where the request body contains `"type": "azure_ai_evaluator"` in `testing_criteria`.
## Suggested Fix
Change the default or routing logic so built-in named evaluators use the supported criteria type (likely `"score_model"` with appropriate prompt configuration, or a new `azure_ai_evaluator`-equivalent type once the API exposes it). All three evaluation paths (`EvaluateAsync`, `EvaluateTracesAsync`, `EvaluateFoundryTargetAsync`) call `FoundryEvalConverter.BuildTestingCriteria` which produces `WireTestingCriterion` instances — fixing the wire model type fixes all paths.
## Environment
| Item | Value |
|---|---|
| Package | `Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1` |
| `Azure.AI.Projects` | `2.1.0-beta.3` |
| Foundry endpoint format | `https://.services.ai.azure.com/api/projects/` |
| .NET | `net10.0` |
| Auth | `DefaultAzureCredential` (working, HTTP 400 is not an auth error) |
## Workaround
Wrap `FoundryEvals` in a try/catch and degrade gracefully to local AgentEval metrics. See [AgentEval samples 12 & 13](https://github.com/AgentEvalHQ/AgentEval) for a reference implementation of this pattern.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.