Lightning-AI / Lightning-AI/LitServe
Websocket Support for Streaming Input and Output
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
----
## 🚀 Feature
Support websocket endpoints to allow two-way real-time data communication.
### Motivation
Currently, the requests are processed with the expectation that the data is complete and stateless. However, the input data isn't always ready immediately for use cases like speech to text, text to speech, audio/speech understanding, especially in time-sensitive situations. With the recent release of Realtime API from OpenAI and a new family of voice AI models (ultravox, mini-omni, llama-omni, moshi), support for streaming input and output could benefit the community in many ways and unlock even more creative uses of AI models.
### Pitch
Support streaming input and output with websocket or any other methods to allow real-time AI applications.
### Alternatives
A typical FastAPI websocket implementation is very template-like:
```python
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
try:
while True:
message = await websocket.receive_bytes()
# process data
# results = model(parse(message))
await websocket.send_json(results)
except WebSocketDisconnect:
logger.error("WebSocket disconnected")
except Exception as e:
logger.error(f"Error: {e}")
if websocket.client_state != WebSocketState.DISCONNECTED:
await websocket.close(code=1001)
finally:
# clean up
```
However, this might make the batching impossible or complicated.
I am new to this repo, so if there is a workaround by hacking the server/spec/api to allow websocket, I am more than happy to contribute. If this is duplicate/irrelevant, sorry for the trouble.
Thanks a million for open sourcing this awesome project. ❤️
### Additional context
Contributor guide
Research direction
Start by reviewing the existing server, spec, and API behavior to determine whether websocket endpoints can fit the current batching model. Compare that flow with the FastAPI websocket example in the issue and define how streaming input, output, disconnects, and errors should work. Done means an agreed implementation path and working real-time streaming support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python
- Domain
- ai, api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100