dotnet / dotnet/extensions

[API Proposal]: Consider populating the `chatResponse.RawRepresentation` after using `updates.ToChatResponse()`

Open
#6,746 0 comments 0 reactions 0 assignees View on GitHub
api-suggestion area-ai
Dominant language
C#
Stars
3.2k
Forks
894
Avg merge
1d 12h
Merged PRs (30d)
23

Description

### Background and motivation

When converting a list of updates into a ChatResponse the `RawRepresentation` of the updates can't be recovered anymore.

This becomes an issue when the caller want to breakglass non-streaming API's that defer their non-streaming implementations to a streaming API.

i.e:

```
OpenAIAssistantChatClient.GetResponseAsync -> GetStreamingAsync().ToChatResponse()
```

This would enable caller breaking glass streaming implementations to be used against non-streaming API's RawRepresentations that are actually a streaming transformation.

One example that is not possible today with the current way is to break glass to retrieve the code generated in the `codeInterpreter` tool from a non-streaming OpenAIAssistantsChatClient API due to this deferrence.

https://github.com/dotnet/extensions/blob/da94b6045fef85f78a5eee183579221cee684e51/src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIAssistantsChatClient.cs#L83

Potential code for capturing the code generated.

```csharp
RunStepDetailsUpdate update

// Process text content
if (update.CodeInterpreterInput != null)
{
if ((update.CodeInterpreterOutputs?.Count ?? 0) > 0)
{
// custom logic to capture the generated code-content

codeContent.Append(update.CodeInterpreterInput);

foreach (var output in update.CodeInterpreterOutputs!)
{
if (output.ImageFileId != null)
{
imageFiles.Add(output.ImageFileId);
}
}
}
}
```

> [!NOTE]
> The Code interpreter output breaking glass is just one example as I'm aware we are going to work on an abstraction to represent HostedCodeInterpreter tool output that could also take a positive potentially use this new API behavior.

### API Proposal

No new API's would be introduced.

### API Usage

```csharp
// Option 1: Expose a list of chatReponseUpdate as the RawRepresentation
Assert.Same(updates, updates.ToChatResponse().RawRepresentation)

// Option 2: Expose a list of the RawRepresentations of the ChatResponseUpdates
Assert.Same(updates[0].RawRepresentation, (updates.ToChatResponse().RawRepresentation as IEnumerable)?.First())
```

### Alternative Designs

_No response_

### Risks

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.