aws / aws/chalice

Support for SSE streaming (OpenAI support)

Open
#2,078 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.1k
Forks
1k
Avg merge
1d 22h
Merged PRs (30d)
2

Description

Hi--we'd like to use Chalice for an OpenAI streaming project (streaming results back from the openai sdk using the streaming flag, ala how ChatGPT streams responses as the AI generates them). It looks like this is supported in the NodeJS runtime, but not so far in the Python runtimes.

There does seem to be a python api in boto to do turn this on: client.lambda.invoke_with_response_stream. Is there some limitation that would stop us from using this in Chalice?

thanks!

Contributor guide

Open the contributing guide

Research direction

Start by investigating Chalice's Python runtimes and how the NodeJS runtime supports SSE streaming. Check whether boto's client.lambda.invoke_with_response_stream can be used for this path and whether the OpenAI SDK's streaming flag fits Chalice's response model. Done means documenting the limitation or defining support for streaming OpenAI results.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.