dotnet / dotnet/extensions

Allow controlling JSON formatting for gen_ai input/output message telemetry

Open
#7,514 1 comment 0 reactions 0 assignees View on GitHub
area-ai enhancement
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

## Summary

`OpenTelemetryChatClient` and related MEAI OpenTelemetry paths currently serialize `gen_ai.input.messages` / `gen_ai.output.messages` as indented JSON, and there does not appear to be a public knob to emit compact JSON instead.

For large prompts, responses, tool calls, tool results, or multimodal content, the extra whitespace increases span attribute size and can make exporter/backend size limits easier to hit.

## Current behavior

The chat path uses `OtelMessageSerializer.DefaultOptions`, which is created by copying `OtelContext.Default.Options`:

```csharp
private static JsonSerializerOptions CreateDefaultOptions()
{
JsonSerializerOptions options = new(OtelContext.Default.Options)
{
Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};

options.TypeInfoResolverChain.Add(AIJsonUtilities.DefaultOptions.TypeInfoResolver!);
options.MakeReadOnly();

return options;
}
```

`OtelContext` is source-generated with `WriteIndented = true`:

```csharp
[JsonSourceGenerationOptions(
PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower,
WriteIndented = true,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
internal sealed partial class OtelContext : JsonSerializerContext;
```

`OtelMessageSerializer.SerializeChatMessages(...)` then serializes the final OTel payload with those options/type metadata:

```csharp
return JsonSerializer.Serialize(output, DefaultOptions.GetTypeInfo(typeof(IList)));
```

`OpenTelemetryChatClient.JsonSerializerOptions` is passed as `customContentSerializerOptions`, but that only influences the fallback serialization path for unknown/custom `AIContent` instances:

```csharp
customContentSerializerOptions?.TryGetTypeInfo(content.GetType(), out JsonTypeInfo? ctsi) is true ? ctsi :
DefaultOptions.TryGetTypeInfo(content.GetType(), out JsonTypeInfo? dtsi) ? dtsi :
null;
```

It does not affect the outer `gen_ai.input.messages` / `gen_ai.output.messages` JSON formatting, nor the built-in OTel message-part shapes.

The realtime path has the same effective behavior: it serializes message payloads through `OtelContext.Default.IEnumerableRealtimeOtelMessage`, which is generated from the same `OtelContext` with `WriteIndented = true`.

## Why this matters

The `gen_ai.input.messages` and `gen_ai.output.messages` attributes can be large, especially when `EnableSensitiveData` is enabled or when `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` is set.

This matters for exporters/backends with size limits. For example, https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/873 discusses Geneva exporter string/event limits:

- Geneva exporter currently has a string field limit of 16 KiB / 16,383 UTF-16 characters.
- Maintainers note there is also an approximately 65,360-byte total event limit from underlying ETW / Linux `user_events` constraints.
- The 16 KiB per-string limit is described as artificial/protective, but the overall event-size limit is not easy to lift.
- Later discussion asks for configurable string-size limits because exceeding the total event size can lead to full event loss.

Given those constraints, always emitting pretty-printed JSON for potentially large GenAI message attributes consumes limited telemetry payload budget without adding semantic value for production pipelines. Compact JSON would reduce the risk of truncation or dropped events/spans and reduce storage/export cost.

## Ask

Please consider adding a supported way to control this formatting. Possible approaches:

1. Add an option on `OpenTelemetryChatClient` / `OpenTelemetryRealtimeClient` to choose compact vs indented OTel message JSON.
2. Add a more general `OpenTelemetry*Client` serialization/formatting option for `gen_ai.input.messages` and `gen_ai.output.messages`.
3. If feasible, make existing `JsonSerializerOptions` influence final OTel message serialization, not only fallback serialization for custom/unknown `AIContent`.

The default could remain unchanged for compatibility. The key ask is to make compact JSON possible without post-processing Activity tags.

## Current workaround

The only practical workaround today is an OpenTelemetry processor that rewrites or removes the message attributes before export, for example by parsing and reserializing `gen_ai.input.messages` / `gen_ai.output.messages` compactly in a `BaseProcessor`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.