Lightning-AI/LitServe

View on GitHub

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

Stars: 3.9k
Forks: 304
Open beginner issues: 0
Indexed issues: 33
Avg merge: 3d 13h
Merged PRs (30d): 6

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 10, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: help wanted good first issue

Beginner friendly Everything

Open All

33 open issues indexed

`fast_queue=True` is silently ignored — the ZMQ transport is never enabled (and would be misconfigured if it were) Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

Lightning-AI/LitServe#751 · Opened Sep 14, 2026
Failed requests leave active_requests permanently incremented Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

Lightning-AI/LitServe#747 · Opened Sep 10, 2026
OpenAISpec rejects standard required and named function tool_choice values Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

Lightning-AI/LitServe#745 · Opened Sep 10, 2026
Sync streaming loop fires AFTER_PREDICT twice and skips decode-request callback events Open

Difficulty 3/5 1-2 days Newbie friendliness 84/100

Lightning-AI/LitServe#735 · Opened Aug 26, 2026
Pass binary payloads to `decode_request` unchanged Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 55/100

Lightning-AI/LitServe#732 · 1 comment · Opened Aug 18, 2026
Prometheus & OpenTelemetry Metrics for LitServe Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/LitServe#724 · 1 comment · Opened Aug 4, 2026
Pass headers to litapi Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Lightning-AI/LitServe#718 · Opened Jul 6, 2026
Example: Serve FunASR/SenseVoice for speech recognition Open

Difficulty 3/5 1-2 days Newbie friendliness 52/100

Lightning-AI/LitServe#688 · 1 comment · Opened Jun 2, 2026
Threads not properly killed on windows Open

bug help wanted

Difficulty 3/5 1-2 days Newbie friendliness 55/100

Lightning-AI/LitServe#684 · Opened Jun 1, 2026
wrap_litserve_start() hangs on requests when worker setup() is slow Open

bug help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

Lightning-AI/LitServe#681 · 1 comment · Opened May 9, 2026
Async task processing via TaskSpec — standalone package, interested in upstreaming? Open

enhancement spec

Difficulty 5/5 Over a week Newbie friendliness 28/100

Lightning-AI/LitServe#668 · 2 comments · 1 reaction · Opened Mar 16, 2026
Swagger UI does not expose request parameters for /predict, causing 500 error when invoked Open

bug help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

Lightning-AI/LitServe#667 · 2 comments · Opened Mar 15, 2026
downloadable documentation from https://lightning.ai/docs/litserve Open

documentation enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/LitServe#630 · 2 reactions · Opened Nov 11, 2025
health probes such as startupz, readyz Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Lightning-AI/LitServe#628 · 6 comments · Opened Oct 29, 2025
Access Health check endpoint for each LitAPI independently Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

Lightning-AI/LitServe#605 · 4 comments · Opened Sep 9, 2025
How can I initialize multiple instances of different model classes on different GPUs? Open

enhancement good first issue help wanted

Difficulty 5/5 Over a week Newbie friendliness 30/100

Lightning-AI/LitServe#572 · 14 comments · Opened Jul 8, 2025
LitServe MCP Bug: tools/call Always Fails Open

bug help wanted mcp

Lightning-AI/LitServe#560 · 6 comments · 1 reaction · 1 assignee · Opened Jun 29, 2025
Iceoryx2 queues Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

Lightning-AI/LitServe#559 · 1 comment · Opened Jun 27, 2025
Optimize Slow CI Tests to Reduce Runtime Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/LitServe#527 · 3 comments · 1 reaction · Opened Jun 4, 2025
start multiple types of models, with any number of instances for each Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/LitServe#514 · 3 comments · Opened May 27, 2025
Multiple endpoints roadmap Open

enhancement

Lightning-AI/LitServe#492 · 4 comments · 6 reactions · 1 assignee · Opened May 13, 2025
Run a mix of async and non-async methods in LitAPI like FastAPI Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/LitServe#490 · 1 reaction · Opened May 13, 2025
Find the latency in async mode Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 25/100

Lightning-AI/LitServe#481 · 1 comment · 1 reaction · Opened May 8, 2025
Support for OpenAI's new Responses API spec Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/LitServe#455 · 7 comments · 2 reactions · Opened Mar 17, 2025
Potential Memory Leak in Response Buffer Open

bug good first issue help wanted

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/LitServe#452 · 7 comments · 1 reaction · Opened Mar 12, 2025
Allow to pass `default_response_class` e.g. `ORJSONResponse` to LitServe server Open

enhancement won't fix

Difficulty 3/5 1-2 days Newbie friendliness 42/100

Lightning-AI/LitServe#436 · 7 comments · 1 reaction · Opened Feb 23, 2025
Allow resource release at the end of life of the LitAPI object Open

enhancement won't fix

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/LitServe#420 · 3 comments · 1 reaction · Opened Feb 3, 2025
Decouple API interface from batching configuration Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Lightning-AI/LitServe#412 · 7 comments · Opened Jan 14, 2025
Support additional content types in post requests Open

enhancement won't fix

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

Lightning-AI/LitServe#362 · 5 comments · 2 reactions · Opened Nov 19, 2024
Websocket Support for Streaming Input and Output Open

enhancement question won't fix

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/LitServe#320 · 15 comments · 3 reactions · Opened Oct 3, 2024
Handle case when Logger.process is stuck Open

enhancement

Lightning-AI/LitServe#289 · 1 assignee · Opened Sep 23, 2024
avoid sending content past the last token during batching / batched streaming Open

won't fix

Lightning-AI/LitServe#57 · 3 comments · 1 reaction · 1 assignee · Opened Apr 23, 2024
auto accelerator for JAX Open

won't fix

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/LitServe#54 · 2 comments · Opened Apr 22, 2024

Lightning-AI/LitServe

Get new issues in your inbox