Azure / Azure/static-web-apps

Linked backends: Streaming does not stream; it arrives in a single payload

Open
#1,180 38 comments 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
346
Forks
67
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

Imagine an Azure Static Web App with a linked backend. The linked backend is an Azure Function App, but based upon our investigations; the issue does not appear to be function app related.

**To Reproduce**

Deploy a Static Web App and a Function App which is a linked backend to the SWA. The backend contains this streaming function named `GetChatCompletionsStream`:

```cs
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Extensions.OpenApi.Core.Attributes;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using System.Net;
using System.Text.Json;
using Azure;
using Azure.AI.OpenAI;
using System.Linq;

namespace ZebraGptFunctionApp.Functions;

public record CompletePromptParameters(
string prompt,
double temperature,
int maxTokens,
string deploymentName,
bool canRecord
);

public class OpenAiFunction
{
private readonly AppSettings _appSettings;
private readonly OpenAIClient _openAIClient;
private readonly ILogger _log;

public OpenAiFunction(
AppSettings appSettings,
ILogger log,
OpenAIClient openAIClient
)
{
_appSettings = appSettings;
_log = log;
_openAIClient = openAIClient;
}

public record ChatCompletionParameters(
string DeploymentName,
float Temperature,
int MaxTokens,
List Messages,
bool CanRecord
);

public record ChatMessageParameter(string Role, string Content);

[FunctionName(nameof(OpenAiFunction.GetChatCompletionsStream))]
public async Task GetChatCompletionsStream(
[HttpTrigger(AuthorizationLevel.Anonymous)] HttpRequest req
)
{
try
{
var chatCompletionParameters = await GetChatCompletionParameters(req);
var chatOptions = GetChatCompletionsOptions(chatCompletionParameters);

var response = req.HttpContext.Response;

response.StatusCode = (int)HttpStatusCode.OK;
// response.ContentType = "text/plain";
response.ContentType = "text/event-stream";

await using var sw = new StreamWriter(response.Body);

var streamingChatCompletionsResponse = await _openAIClient.Client.GetChatCompletionsStreamingAsync(
deploymentOrModelName: chatCompletionParameters.DeploymentName,
chatOptions
);

using StreamingChatCompletions streamingChatCompletions = streamingChatCompletionsResponse.Value;

await foreach (StreamingChatChoice choice in streamingChatCompletions.GetChoicesStreaming())
{
await foreach (ChatMessage message in choice.GetMessageStreaming())
{
// THIS IS STREAMING
await sw.WriteAsync(message.Content);
await sw.FlushAsync();
}
}
}
catch (Exception ex)
{
_log.LogError(ex, "Problem getting chat completion");
req.HttpContext.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
}

// Required to avoid the host trying to add headers to the response
return new EmptyResult();
}

private async static Task GetChatCompletionParameters(HttpRequest req)
{
var content = await new StreamReader(req.Body).ReadToEndAsync();

var chatCompletionParameters =
JsonSerializer.Deserialize(content, new JsonSerializerOptions
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
})!;

return chatCompletionParameters;
}

private static ChatCompletionsOptions GetChatCompletionsOptions(ChatCompletionParameters chatCompletionParameters)
{
var chatRoleMapping = new Dictionary()
{
{ "system", ChatRole.System }, { "user", ChatRole.User }, { "assistant", ChatRole.Assistant }
};

var chatOptions = new ChatCompletionsOptions()
{
Temperature = chatCompletionParameters.Temperature,
MaxTokens = chatCompletionParameters.MaxTokens,
NucleusSamplingFactor = (float)0.95,
FrequencyPenalty = 0,
PresencePenalty = 0,
};

// it has to be added like this because by design Messages is a readonly. https://github.com/Azure/azure-sdk-for-net/issues/35096
chatCompletionParameters.Messages.ForEach(message =>
chatOptions.Messages.Add(new ChatMessage(chatRoleMapping[message.Role], message.Content)));

return chatOptions;
}
}
```

Eagle peeps will note we're building an Open AI chat mechanism - but what's significant here is we stream text to the caller.

On the front end we have a TypeScript function that looks like this:

```ts
try {
if (!userInput.trim()) {
return;
}

const userMessage: ChatMessage = {
role: "user",
content: userInput,
};
setMessages((prevMessages) => [...prevMessages, userMessage]);
setUserInput("");
setIsLoading(true);

const response = await fetch("/api/GetChatCompletionsStream", {
method: "POST",
body: JSON.stringify({
deploymentName: "OpenAi-gpt-35-turbo",
maxTokens,
temperature,
messages: [...messages, userMessage],
} as ChatCompletionParameters),
headers: {
Accept: "text/event-stream",
"Content-Type": "application/json",
},
});

if (!response.body) {
return;
}

const reader = response.body.getReader();

let reads = 0;
let responseMessage = "";
// eslint-disable-next-line no-constant-condition
while (true) {
const { done, value } = await reader.read();

if (done) {
break;
}

const fragment = new TextDecoder("utf-8").decode(value);
console.log(`fragment #${++reads}`, fragment);
responseMessage += fragment;
setLiveResponseMessage(responseMessage);
}
console.log(`done in ${reads} reads`);
setMessages((prevMessages) => [
...prevMessages,
{ role: "assistant", content: responseMessage },
]);
setIsLoading(false);
setLiveResponseMessage("");
} catch (error) {
console.error("Error sending message:", error);
}
```

**Expected behavior**

Running locally, this works as expected: streaming. Deployed - it does not; we get a *single* payload in our "stream".

**Screenshots**

Running locally (it working):

image

Deployed to Azure (it not working):

image

Notice the `1 reads` - that's streaming not working.

**Additional context**
We have tried directly accessing the function app from the front end of the static web app and confirmed streaming from the function app directly works. However, using this approach we lose all the benefits of linked backends.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.