Lightning-AI / Lightning-AI/LitServe
Optimize Slow CI Tests to Reduce Runtime
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 304
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 6
Description
## Optimize Slow CI Tests to Reduce Runtime
**Description:**
Our CI tests are hitting timeout limits due to some slow tests and increased test count. Increasing timeouts helps temporarily, but we need to optimize tests for faster CI runs.
> Follow up to #526
**What to do:**
* [Identify and analyze slow tests](https://github.com/Lightning-AI/LitServe/pull/526#issuecomment-2939231561) (e.g., `test_server_with_disabled_timeout`, `test_readme`).
* Refactor or mock heavy dependencies to speed them up.
* Improve overall CI runtime for quicker feedback.
Slowest 100 durations
```
============================ slowest 100 durations =============================
15.61s call tests/test_simple.py::test_server_with_disabled_timeout[False]
13.62s call tests/test_simple.py::test_server_with_disabled_timeout[True]
10.02s call tests/test_readme.py::test_readme
10.01s call tests/test_transport.py::TestTransportIntegration::test_send_receive_cycle
9.27s call tests/test_lit_server.py::test_inject_context
7.14s call tests/e2e/test_e2e.py::test_e2e_openai_embedding_with_batching
7.09s call tests/test_simple.py::test_workers_health_with_custom_health_method[True]
7.09s call tests/test_simple.py::test_workers_health_with_custom_health_method[False]
6.56s call tests/test_callbacks.py::test_request_tracker
6.39s call tests/test_lit_server.py::test_concurrent_async_inference[100]
6.38s call tests/test_lit_server.py::test_concurrent_async_inference[50]
6.30s call tests/test_lit_server.py::test_concurrent_async_inference[10]
5.71s call tests/e2e/test_e2e.py::test_e2e_openai_with_batching
5.58s call tests/test_lit_server.py::test_http_exception
5.46s call tests/e2e/test_e2e.py::test_openai_parity
5.33s call tests/e2e/test_e2e.py::test_run
5.31s call tests/e2e/test_e2e.py::test_run_with_port
5.16s call tests/e2e/test_e2e.py::test_e2e_default_batching
5.13s call tests/e2e/test_e2e.py::test_e2e_batched_streaming
5.09s call tests/test_simple.py::test_workers_health_custom_path[False]
5.09s call tests/test_simple.py::test_workers_health_custom_path[True]
5.08s call tests/test_simple.py::test_workers_health[True]
5.08s call tests/e2e/test_e2e.py::test_openai_embedding_parity
5.08s call tests/test_simple.py::test_workers_health[False]
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_tools
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_audio_input
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_image_input
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_response_format
5.03s call tests/e2e/test_e2e.py::test_e2e_default_spec
5.02s call tests/e2e/test_e2e.py::test_e2e_default_async_streaming
5.02s call tests/e2e/test_e2e.py::test_e2e_default_api
5.02s call tests/e2e/test_e2e.py::test_e2e_single_streaming
```
**Goal:**
Faster, more reliable CI without needing longer timeouts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the slow-test report and the analysis in PR #526, focusing on tests/test_simple.py::test_server_with_disabled_timeout and tests/test_readme.py::test_readme. Compare the other listed tests in tests/test_transport.py, tests/test_lit_server.py, tests/test_callbacks.py, and tests/e2e/test_e2e.py to identify expensive dependencies or waits. Done means the affected tests retain coverage and CI completes reliably without increased timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, performance, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100