Lightning-AI / Lightning-AI/LitServe
Multiple endpoints roadmap
@aniketmaurya is already working on this.
Since May 13, 2025.
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
## 🚀 Support for Multiple Endpoints in LitServe
We are introducing support for multiple endpoints in **LitServe**, which will require minor architectural adjustments to better separate concerns between the core components.
### ✨ Proposed Usage
Here’s an example of what the new API could look like:
```python
api1 = VisionEncoderAPI(batch_size=4, api_path="/ap1")
api2 = TextEncoderAPI(batch_size=8, api_path="/path2)
def resize(...):
...
server = LitServer([
api1,
api2,
{"/random": resize}
])
server.run()
```
### 🔍 Design Goals
To cleanly support this functionality, we need to ensure that responsibilities are clearly divided between the two main entities:
#### `LitAPI`
- Manages properties that pertain to the logic of individual APIs, such as:
- `batch_size`
- `loop`
- `stream`
- `api_path`
#### `LitServer`
- Handles server-level configuration, such as:
- Endpoint routing
#### Shared Responsibility (`LitAPI` + `LitServer`)
- Callback handling
- Streaming functionality
### 📌 Action Items
In [PR #468](https://github.com/Lightning-AI/LitServe/pull/468), we successfully moved the `batch_size` parameter from `LitServer` to `LitAPI` to align with this separation of concerns. We need to apply a similar refactoring for other attributes:
- [x] Move `loop` from `LitServer` to `LitAPI` - https://github.com/Lightning-AI/LitServe/pull/512
- [x] Move `stream` argument from LitServer to LitAPI - https://github.com/Lightning-AI/LitServe/pull/512
- [x] Move `api_path` - https://github.com/Lightning-AI/LitServe/pull/512
- [x] Remove `LitServer` argument from `LitSpec.setup`. Connect the queues and response buffer with a connector instead.
- [x] Add multiple LitAPI support - https://github.com/Lightning-AI/LitServe/pull/513
- [x] Allow streaming and non-streaming together. - https://github.com/Lightning-AI/LitServe/pull/518
- [ ] Scale multiple APIs separately as described [here](https://github.com/Lightning-AI/LitServe/issues/514).
- [ ] Multiple endpoints with a single LitAPI (shared setup)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.