microsoft / microsoft/agent-framework

.NET: [Feature]: FoundryEvals — uploaded (file-id) datasets and per-criterion judge configuration

Open
#7,364 0 comments 0 reactions 1 assignee Claimed by @chetantoshniwal View on GitHub
.NET
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

### Description

Two independent widenings of the `FoundryEvals` client surface (`Microsoft.Agents.AI.Foundry`, `main` @ `7b6d25798`). Both are supported by the underlying Evals API (`evals.create` / `evals.runs.create`) but not reachable through the client today.

First, credit where due: #6267 already removed the biggest client-side ceiling here — `FoundryEvalConverter.ResolveEvaluator` now passes any `builtin.*`-prefixed name through to the service instead of rejecting names missing from the client dictionary, so new server-side evaluators are usable without a package update. The two asks below are the remainder.

**1. No uploaded (file-id) dataset source.**

The wire layer materializes every eval row inline: `WireJsonlDataSource` only carries a `WireFileContentSource` (`type: "file_content"`), so each run re-uploads its dataset as inline JSONL content. The Evals API also accepts previously uploaded JSONL datasets referenced by file id. Large or shared eval sets shouldn't have to be re-materialized per run.

Proposal: an overload (or data-source parameter) that accepts a file id, mapping to the API's file-based data source alongside the existing inline path.

**2. One judge model for all criteria, no per-criterion threshold.**

Every `FoundryEvals` constructor takes a single `string model` that configures the judge for all criteria, and no threshold is expressible anywhere in the client surface. The API allows per-criterion configuration. `FoundryEvaluatorSpec` (introduced in #6267) is currently `BuiltinName | GeneratedEvaluatorRef` — it is the natural carrier for per-criterion settings.

Proposal: optional per-spec configuration, e.g.:

```csharp
public readonly struct FoundryEvaluatorSpec
{
// existing: BuiltinName, GeneratedRef, ...
public string? Model { get; init; } // overrides the FoundryEvals-level judge model
public double? Threshold { get; init; } // per-criterion pass threshold
}
```

- **What problem does it solve?** Removes the remaining client-side ceilings on server capability: big datasets without inline re-upload, and mixed judge models / thresholds per criterion in a single eval.
- **Expected behavior:** existing calls unchanged; both additions are opt-in and additive.
- **Alternatives considered:** hand-rolling raw `evals.create` payloads next to `FoundryEvals` (duplicates the wire layer); running one `FoundryEvals` instance per judge model (loses the single-eval grouping and duplicates runs).

### Code Sample

```csharp
FoundryEvals evals = new(
projectClient,
model: "gpt-4o",
new FoundryEvaluatorSpec("relevance") { Threshold = 4.0 },
new FoundryEvaluatorSpec("builtin.coherence") { Model = "gpt-4o-mini" });

// uploaded-dataset run instead of inline rows (shape illustrative):
await evals.EvaluateDatasetAsync(fileId: uploadedJsonlFileId);
```

### Language/SDK

.NET

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.