dotnet / dotnet/extensions

OpenAIResponsesChatClient drops content-level RawRepresentation for System and Developer messages

Open
#7,679 0 comments 0 reactions 0 assignees View on GitHub
untriaged
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

## Description

`OpenAIResponsesChatClient` discards content-level `AIContent.RawRepresentation` for `System` and `Developer` messages, while preserving it for `User` messages.

In [`ToOpenAIResponseItems`](https://github.com/dotnet/extensions/blob/49b5c83df8b280b6e8a113d0cf1df2407dbbb73c/src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIResponsesChatClient.cs#L1283-L1303) the system/developer branch flattens the message to a single string and rebuilds the item from it:

```csharp
if (input.Role == ChatRole.System ||
input.Role == OpenAIClientExtensions.ChatRoleDeveloper)
{
string text = input.Text;
if (!string.IsNullOrWhiteSpace(text))
{
yield return input.Role == ChatRole.System ?
ResponseItem.CreateSystemMessageItem(text) :
ResponseItem.CreateDeveloperMessageItem(text);
}

continue;
}
```

Because the item is reconstructed from `input.Text`, any `ResponseContentPart` a caller attached as `RawRepresentation` on the message's contents is dropped. The `User` branch immediately below does honour it, both for contents that map to a whole `ResponseItem` (`{ RawRepresentation: ResponseItem rawRep } => rawRep`) and for contents that map to `ResponseContentPart`s grouped via `ResponseItem.CreateUserMessageItem(parts)`.

The failure is silent: no exception is thrown, the request succeeds, and the customization simply is not on the wire.

## Why this matters

`RawRepresentation` is the documented escape hatch for provider-specific fields that `Microsoft.Extensions.AI` does not model. Setting it on a content part is the only supported way to add a property to an `input_text` part.

The concrete case that led us here is OpenAI's **explicit prompt caching for GPT-5.6**, which requires a `prompt_cache_breakpoint` property on a content part:

```json
{
"type": "input_text",
"text": "...",
"prompt_cache_breakpoint": { "mode": "explicit" }
}
```

Under `prompt_cache_options.mode = "explicit"` nothing is cached unless a content part carries that marker, and a breakpoint caches everything up to and including the part it sits on. The natural placement for the first breakpoint is the end of the system prompt. That is currently impossible through MEAI — the property is stripped before serialization, and the request caches nothing.

The service itself accepts a breakpoint on a system message: sending the same body by hand to `/openai/v1/responses` on a `gpt-5.6-terra` deployment produced `cached_tokens: 9106`. Only the MEAI conversion path loses it.

## Reproduction

Minimal console app, `net10.0`, referencing `Microsoft.Extensions.AI.OpenAI` `10.6.0` and `OpenAI` `2.10.0`. A capturing `HttpMessageHandler` records the outgoing body; no network access or API key is needed.

```csharp
using System.ClientModel;
using System.ClientModel.Primitives;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenAI.Responses;

class CapturingHandler : HttpMessageHandler
{
public string? Body;
protected override async Task SendAsync(HttpRequestMessage req, CancellationToken ct)
{
Body = req.Content is null ? "" : await req.Content.ReadAsStringAsync(ct);
return new HttpResponseMessage(System.Net.HttpStatusCode.OK)
{
Content = new StringContent(
"""{"id":"r","object":"response","created_at":0,"status":"completed","model":"m","output":[],"usage":{"input_tokens":1,"output_tokens":1,"total_tokens":2}}""",
Encoding.UTF8, "application/json"),
};
}
}

static class Program
{
// Adds an arbitrary property to a ResponseContentPart by round-tripping its JSON,
// because IJsonModel.Create replaces rather than merges.
static T AugmentJsonModel(T model, string key, object value) where T : IJsonModel
{
using var doc = JsonDocument.Parse(ModelReaderWriter.Write(model, ModelReaderWriterOptions.Json));
using var ms = new MemoryStream();
using (var writer = new Utf8JsonWriter(ms))
{
writer.WriteStartObject();
foreach (var p in doc.RootElement.EnumerateObject())
{
p.WriteTo(writer);
}

writer.WritePropertyName(key);
JsonSerializer.Serialize(writer, value);
writer.WriteEndObject();
}

return ModelReaderWriter.Read(new BinaryData(ms.ToArray()), ModelReaderWriterOptions.Json)!;
}

static TextContent Marked(string text)
{
var part = AugmentJsonModel(
ResponseContentPart.CreateInputTextPart(text),
"prompt_cache_breakpoint",
new Dictionary { ["mode"] = "explicit" });

return new TextContent(text) { RawRepresentation = part };
}

static async Task Main()
{
var handler = new CapturingHandler();
var options = new OpenAIClientOptions
{
Endpoint = new Uri("https://example.invalid/openai/v1"),
Transport = new HttpClientPipelineTransport(new HttpClient(handler)),
};

IChatClient client = new OpenAIClient(new ApiKeyCredential("k"), options)
.GetResponsesClient()
.AsIChatClient("gpt-5.6-terra");

await client.GetResponseAsync(
[
new ChatMessage(ChatRole.System, new List { Marked("SYSTEM") }),
new ChatMessage(ChatRole.User, new List { Marked("USER") }),
]);

Console.WriteLine(handler.Body);
}
}
```

### Actual output

```json
{"model":"gpt-5.6-terra","input":[
{"type":"message","role":"system","content":[{"type":"input_text","text":"SYSTEM"}]},
{"type":"message","role":"user","content":[{"type":"input_text","text":"USER","prompt_cache_breakpoint":{"mode":"explicit"}}]}
]}
```

Both messages had an identical `RawRepresentation` attached. It survives on `user` and is gone on `system`.

The same happens for `ChatRoleDeveloper`.

### Expected output

`prompt_cache_breakpoint` present on the `system` content part as well.

## Suggested fix

Have the system/developer branch reuse the caller's `ResponseContentPart` when one is present, instead of unconditionally rebuilding from `input.Text` — mirroring the `parts` handling already used for user messages. Something along the lines of collecting parts from `input.Contents` (falling back to `CreateInputTextPart` when `RawRepresentation` is not a `ResponseContentPart`) and passing them to the system/developer item factory.

Note that flattening to `input.Text` also silently coalesces multiple `TextContent` items into one part, which may be worth preserving separately.

## Workaround

Insert a synthetic `user` message containing `"."` immediately after the system prompt and put the breakpoint on that instead, since the user path preserves `RawRepresentation`. This works but adds a real token to every request and puts a spurious turn in the conversation.

## Environment

| | |
|---|---|
| `Microsoft.Extensions.AI` | 10.6.0 |
| `Microsoft.Extensions.AI.Abstractions` | 10.6.0 |
| `Microsoft.Extensions.AI.OpenAI` | 10.6.0 |
| `OpenAI` | 2.10.0 |
| TFM | net10.0 |
| OS | macOS (arm64) |

Also reproduces against Azure OpenAI (`*.openai.azure.com`) via `AzureOpenAIClient`, since the conversion is shared.

Contributor guide

Open the contributing guide

Research direction

Start in src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIResponsesChatClient.cs at ToOpenAIResponseItems, especially the system/developer branch around lines 1283-1303, and compare it with the user branch immediately below. Use the provided capturing-handler reproduction to verify that system and developer content-level RawRepresentation reaches the outgoing request, while preserving the expected fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
api, backend-api-design
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.