dotnet / dotnet/extensions

[AI Evaluation] DiskBasedResultStore crashes serializing EvaluationContext with multiple AIContent items

Open
#7,592 2 comments 0 reactions 0 assignees View on GitHub
untriaged
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

### Description

`DiskBasedResultStore.WriteResultsAsync()` crashes with `System.NotSupportedException` when an `EvaluationContext` contains multiple `AIContent` items in its `Contents` property. This affects `RetrievalEvaluator`, `BLEUEvaluator`, `GLEUEvaluator`, and `F1Evaluator` — any evaluator whose context stores multiple text items.

The root cause: `JsonUtilities+JsonContext` (the source-generated JSON context in `Microsoft.Extensions.AI.Evaluation.Reporting`) only registers `ScenarioRunResult`, `Dataset`, and `CacheEntry` as serializable types. When the custom `EvaluationContextConverter` tries to serialize the inherited `Contents` property (type `IReadOnlyList`), the source-generated context has no metadata for that type.

### Reproduction Steps

1. Create a file-based .NET app with the following single-file repro (requires .NET 10 SDK):

```csharp
#:property TargetFramework=net10.0
#:property NoWarn=AIEVAL001
#:package Microsoft.Extensions.AI.Evaluation@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Quality@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Reporting@10.7.0
#:package Microsoft.Extensions.AI.OpenAI@10.7.0
#:package OpenAI@2.11.0

using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI.Evaluation.Reporting;
using Microsoft.Extensions.AI.Evaluation.Reporting.Storage;
using OpenAI;

if (args.Length < 3)
{
Console.Error.WriteLine("Usage: dotnet run --file repro.cs ");
return 1;
}

var client = new OpenAIClient(
new System.ClientModel.ApiKeyCredential(args[2]),
new OpenAIClientOptions { Endpoint = new Uri(args[0]) });

var chatClient = client.GetChatClient(args[1]).AsIChatClient();
var chatConfig = new ChatConfiguration(chatClient);

var evaluators = new IEvaluator[] { new RetrievalEvaluator() };

var config = DiskBasedReportingConfiguration.Create(
storageRootPath: "./repro-results",
evaluators: evaluators,
chatConfiguration: chatConfig,
enableResponseCaching: false,
executionName: "repro");

await using var run = await config.CreateScenarioRunAsync("repro.scenario", iterationName: "1");

var messages = new List
{
new(ChatRole.User, "How many moons does Saturn have?")
};

var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, "Saturn has 146 confirmed moons."));

var context = new RetrievalEvaluatorContext(
"Saturn has 146 confirmed moons as of 2023.",
"Jupiter has 95 known moons.");

Console.WriteLine("Evaluating...");
var result = await run.EvaluateAsync(
messages,
response,
additionalContext: [context]);

var metric = result.Metrics.Values.OfType().Single();
Console.WriteLine($"Score: {metric.Value}, Failed: {metric.Interpretation?.Failed}");

// Crash happens when DisposeAsync serializes results to disk
```

2. Run: `dotnet run --file repro.cs `

### Expected behavior

`DiskBasedResultStore` should successfully serialize `EvaluationContext` objects regardless of how many `AIContent` items are in the `Contents` list. The evaluation result should be persisted to disk without errors.

### Actual behavior

Evaluation succeeds (score produced, metrics returned), but `ScenarioRun.DisposeAsync()` crashes when `DiskBasedResultStore.WriteResultsAsync()` tries to serialize the `EvaluationContext`:

```
Unhandled exception. System.NotSupportedException: JsonTypeInfo metadata for type
'System.Collections.Generic.IReadOnlyList`1[Microsoft.Extensions.AI.AIContent]' was not
provided by TypeInfoResolver of type
'[Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.JsonUtilities+JsonContext,
Microsoft.Extensions.AI.AIJsonUtilities+JsonContext]'. The unsupported member type is
located on type 'Microsoft.Extensions.AI.Evaluation.EvaluationContext'.
Path: $.EvaluationResult.Metrics.Context.

at System.Text.Json.ThrowHelper.ThrowNotSupportedException_NoMetadataForType(...)
at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(...)
at Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.EvaluationContextConverter.Write(...)
...
at Microsoft.Extensions.AI.Evaluation.Reporting.Storage.DiskBasedResultStore.WriteResultsAsync(...)
at Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRun.DisposeAsync()
```

### Affected evaluators

| Evaluator | Context Contents | Result |
|-----------|-----------------|--------|
| `GroundednessEvaluator` | 1 `TextContent` | Works |
| `EquivalenceEvaluator` | 1 `TextContent` | Works |
| `RetrievalEvaluator` | N `TextContent` (chunks) | Crashes |
| `BLEUEvaluator` | N `TextContent` (references) | Crashes |
| `GLEUEvaluator` | N `TextContent` (references) | Crashes |
| `RelevanceTruthAndCompletenessEvaluator` | No context | Works |

### Configuration

- .NET SDK 10.0.301
- `Microsoft.Extensions.AI.Evaluation.Reporting` 10.7.0
- `Microsoft.Extensions.AI.Evaluation.Quality` 10.7.0
- Windows 11 x64
- Reproduces with any OpenAI-compatible endpoint (Azure OpenAI, OpenCode, etc.)

### Other information

Likely fix: add `[JsonSerializable(typeof(IReadOnlyList))]` to the `JsonContext` in `src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/JsonSerialization/JsonUtilities.cs`, or update `EvaluationContextConverter.Write()` to handle multi-item `Contents` without relying on source-generated metadata.

Contributor guide

Open the contributing guide

Research direction

Start with src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/JsonSerialization/JsonUtilities.cs and inspect EvaluationContextConverter.Write(), then run the supplied .NET 10 reproduction. Done means DiskBasedResultStore persists an EvaluationContext containing multiple AIContent items without ScenarioRun.DisposeAsync() throwing.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.