Azure / Azure/aoai-apim

Handling Token calculation from APIM for SSE Events

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
116
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Following [this guidelines for sse in APIM](https://learn.microsoft.com/en-us/azure/api-management/how-to-server-sent-events#guidelines-for-sse) I'm trying to implement event hub logging to send the response of SSE events - *Azure OpenAI Streaming API* as I would like to [calculate the tokens](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb), in my case there are multiple different clients/users and want to do this calculation on my side(APIM) so that I can tell the number of tokens are consumed by different clients/users.

The current policy will look like below.

```xml
@{return new JObject(
new JProperty("records", new JArray() {
new JObject(
new JProperty("EventTime", DateTime.UtcNow.ToString()),
new JProperty("ServiceName", context.Deployment.ServiceName),
new JProperty("RequestId", context.RequestId),
new JProperty("RequestIp", context.Request.IpAddress),
new JProperty("OperationName", context.Operation.Name),
new JProperty("response", context.Response.Body.As(preserveContent: true).ToString())
)
})).ToString();}
```

During the debug I could see it took a good amount of time to read the response payload (for sending to event hub)

```
{
"source": "response-payload-handler",
"timestamp": "2023-10-03T08:03:59.4607171Z",
"elapsed": "00:00:01.9290785",
"data": "Reading response payload because next handler requires access to it."
},
{
"source": "response-payload-handler",
"timestamp": "2023-10-03T08:04:05.1387406Z",
"elapsed": "00:00:07.6014120",
"data": "Reading response payload succeeded. 131316 bytes read."
},
```

If I remove below line
```
new JProperty("response", context.Response.Body.As(preserveContent: true).ToString()))
```

I don't see the `repose-payload-handler` task that is consuming a lot of time earlier. I also think due to implementing the eventhub for sending the response body, I suspect the streaming is not happening (checked from postman) It wait to give the full response till it completes the eventhub

Question is: Is there any way I could achieve this streaming (SSE Events) to log the response body async somewhere (event hub or servicebus ) without affecting the original streaming capability?

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files or tests are named. Start by reproducing the APIM policy with the log-to-eventhub policy and context.Response.Body.As(preserveContent: true), then compare SSE timing with and without response logging; done means the response streams without waiting for event-hub logging while the response data remains available for token calculation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
api, cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.