[AI Evaluation] DiskBasedResultStore crashes serializing EvaluationContext with multiple AIContent items
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
### Description
`DiskBasedResultStore.WriteResultsAsync()` crashes with `System.NotSupportedException` when an `EvaluationContext` contains multiple `AIContent` items in its `Contents` property. This affects `RetrievalEvaluator`, `BLEUEvaluator`, `GLEUEvaluator`, and `F1Evaluator` — any evaluator whose context stores multiple text items.
The root cause: `JsonUtilities+JsonContext` (the source-generated JSON context in `Microsoft.Extensions.AI.Evaluation.Reporting`) only registers `ScenarioRunResult`, `Dataset`, and `CacheEntry` as serializable types. When the custom `EvaluationContextConverter` tries to serialize the inherited `Contents` property (type `IReadOnlyList`), the source-generated context has no metadata for that type.
### Reproduction Steps
1. Create a file-based .NET app with the following single-file repro (requires .NET 10 SDK):
```csharp
#:property TargetFramework=net10.0
#:property NoWarn=AIEVAL001
#:package Microsoft.Extensions.AI.Evaluation@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Quality@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Reporting@10.7.0
#:package Microsoft.Extensions.AI.OpenAI@10.7.0
#:package OpenAI@2.11.0
using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI.Evaluation.Reporting;
using Microsoft.Extensions.AI.Evaluation.Reporting.Storage;
using OpenAI;
if (args.Length < 3)
{
Console.Error.WriteLine("Usage: dotnet run --file repro.cs ");
return 1;
}
var client = new OpenAIClient(
new System.ClientModel.ApiKeyCredential(args[2]),
new OpenAIClientOptions { Endpoint = new Uri(args[0]) });
var chatClient = client.GetChatClient(args[1]).AsIChatClient();
var chatConfig = new ChatConfiguration(chatClient);
var evaluators = new IEvaluator[] { new RetrievalEvaluator() };
var config = DiskBasedReportingConfiguration.Create(
storageRootPath: "./repro-results",
evaluators: evaluators,
chatConfiguration: chatConfig,
enableResponseCaching: false,
executionName: "repro");
await using var run = await config.CreateScenarioRunAsync("repro.scenario", iterationName: "1");
var messages = new List
{
new(ChatRole.User, "How many moons does Saturn have?")
};
var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, "Saturn has 146 confirmed moons."));
var context = new RetrievalEvaluatorContext(
"Saturn has 146 confirmed moons as of 2023.",
"Jupiter has 95 known moons.");
Console.WriteLine("Evaluating...");
var result = await run.EvaluateAsync(
messages,
response,
additionalContext: [context]);
var metric = result.Metrics.Values.OfType().Single();
Console.WriteLine($"Score: {metric.Value}, Failed: {metric.Interpretation?.Failed}");
// Crash happens when DisposeAsync serializes results to disk
```
2. Run: `dotnet run --file repro.cs `
### Expected behavior
`DiskBasedResultStore` should successfully serialize `EvaluationContext` objects regardless of how many `AIContent` items are in the `Contents` list. The evaluation result should be persisted to disk without errors.
### Actual behavior
Evaluation succeeds (score produced, metrics returned), but `ScenarioRun.DisposeAsync()` crashes when `DiskBasedResultStore.WriteResultsAsync()` tries to serialize the `EvaluationContext`:
```
Unhandled exception. System.NotSupportedException: JsonTypeInfo metadata for type
'System.Collections.Generic.IReadOnlyList`1[Microsoft.Extensions.AI.AIContent]' was not
provided by TypeInfoResolver of type
'[Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.JsonUtilities+JsonContext,
Microsoft.Extensions.AI.AIJsonUtilities+JsonContext]'. The unsupported member type is
located on type 'Microsoft.Extensions.AI.Evaluation.EvaluationContext'.
Path: $.EvaluationResult.Metrics.Context.
at System.Text.Json.ThrowHelper.ThrowNotSupportedException_NoMetadataForType(...)
at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(...)
at Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.EvaluationContextConverter.Write(...)
...
at Microsoft.Extensions.AI.Evaluation.Reporting.Storage.DiskBasedResultStore.WriteResultsAsync(...)
at Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRun.DisposeAsync()
```
### Affected evaluators
| Evaluator | Context Contents | Result |
|-----------|-----------------|--------|
| `GroundednessEvaluator` | 1 `TextContent` | Works |
| `EquivalenceEvaluator` | 1 `TextContent` | Works |
| `RetrievalEvaluator` | N `TextContent` (chunks) | Crashes |
| `BLEUEvaluator` | N `TextContent` (references) | Crashes |
| `GLEUEvaluator` | N `TextContent` (references) | Crashes |
| `RelevanceTruthAndCompletenessEvaluator` | No context | Works |
### Configuration
- .NET SDK 10.0.301
- `Microsoft.Extensions.AI.Evaluation.Reporting` 10.7.0
- `Microsoft.Extensions.AI.Evaluation.Quality` 10.7.0
- Windows 11 x64
- Reproduces with any OpenAI-compatible endpoint (Azure OpenAI, OpenCode, etc.)
### Other information
Likely fix: add `[JsonSerializable(typeof(IReadOnlyList))]` to the `JsonContext` in `src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/JsonSerialization/JsonUtilities.cs`, or update `EvaluationContextConverter.Write()` to handle multi-item `Contents` without relying on source-generated metadata.
Contributor guide
Research direction
Start with src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/JsonSerialization/JsonUtilities.cs and inspect EvaluationContextConverter.Write(), then run the supplied .NET 10 reproduction. Done means DiskBasedResultStore persists an EvaluationContext containing multiple AIContent items without ScenarioRun.DisposeAsync() throwing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100