Lightning-AI / Lightning-AI/LitServe

Websocket Support for Streaming Input and Output

Open
#320 15 comments 3 reactions 0 assignees View on GitHub
enhancement question won't fix
Dominant language
Python
Stars
3.9k
Forks
304
Avg merge
3d 13h
Merged PRs (30d)
6

Description

----

## 🚀 Feature

Support websocket endpoints to allow two-way real-time data communication.

### Motivation

Currently, the requests are processed with the expectation that the data is complete and stateless. However, the input data isn't always ready immediately for use cases like speech to text, text to speech, audio/speech understanding, especially in time-sensitive situations. With the recent release of Realtime API from OpenAI and a new family of voice AI models (ultravox, mini-omni, llama-omni, moshi), support for streaming input and output could benefit the community in many ways and unlock even more creative uses of AI models.

### Pitch

Support streaming input and output with websocket or any other methods to allow real-time AI applications.

### Alternatives

A typical FastAPI websocket implementation is very template-like:

```python
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
try:
while True:
message = await websocket.receive_bytes()
# process data
# results = model(parse(message))
await websocket.send_json(results)
except WebSocketDisconnect:
logger.error("WebSocket disconnected")
except Exception as e:
logger.error(f"Error: {e}")
if websocket.client_state != WebSocketState.DISCONNECTED:
await websocket.close(code=1001)
finally:
# clean up
```

However, this might make the batching impossible or complicated.

I am new to this repo, so if there is a workaround by hacking the server/spec/api to allow websocket, I am more than happy to contribute. If this is duplicate/irrelevant, sorry for the trouble.

Thanks a million for open sourcing this awesome project. ❤️

### Additional context

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing server, spec, and API behavior to determine whether websocket endpoints can fit the current batching model. Compare that flow with the FastAPI websocket example in the issue and define how streaming input, output, disconnects, and errors should work. Done means an agreed implementation path and working real-time streaming support.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
ai, api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.