Lightning-AI / Lightning-AI/LitServe

Optimize Slow CI Tests to Reduce Runtime

Open
#527 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Python
Stars
3.9k
Forks
304
Avg merge
3d 13h
Merged PRs (30d)
6

Description

## Optimize Slow CI Tests to Reduce Runtime

**Description:**
Our CI tests are hitting timeout limits due to some slow tests and increased test count. Increasing timeouts helps temporarily, but we need to optimize tests for faster CI runs.
> Follow up to #526

**What to do:**

* [Identify and analyze slow tests](https://github.com/Lightning-AI/LitServe/pull/526#issuecomment-2939231561) (e.g., `test_server_with_disabled_timeout`, `test_readme`).
* Refactor or mock heavy dependencies to speed them up.
* Improve overall CI runtime for quicker feedback.

Slowest 100 durations

```
============================ slowest 100 durations =============================
15.61s call tests/test_simple.py::test_server_with_disabled_timeout[False]
13.62s call tests/test_simple.py::test_server_with_disabled_timeout[True]
10.02s call tests/test_readme.py::test_readme
10.01s call tests/test_transport.py::TestTransportIntegration::test_send_receive_cycle
9.27s call tests/test_lit_server.py::test_inject_context
7.14s call tests/e2e/test_e2e.py::test_e2e_openai_embedding_with_batching
7.09s call tests/test_simple.py::test_workers_health_with_custom_health_method[True]
7.09s call tests/test_simple.py::test_workers_health_with_custom_health_method[False]
6.56s call tests/test_callbacks.py::test_request_tracker
6.39s call tests/test_lit_server.py::test_concurrent_async_inference[100]
6.38s call tests/test_lit_server.py::test_concurrent_async_inference[50]
6.30s call tests/test_lit_server.py::test_concurrent_async_inference[10]
5.71s call tests/e2e/test_e2e.py::test_e2e_openai_with_batching
5.58s call tests/test_lit_server.py::test_http_exception
5.46s call tests/e2e/test_e2e.py::test_openai_parity
5.33s call tests/e2e/test_e2e.py::test_run
5.31s call tests/e2e/test_e2e.py::test_run_with_port
5.16s call tests/e2e/test_e2e.py::test_e2e_default_batching
5.13s call tests/e2e/test_e2e.py::test_e2e_batched_streaming
5.09s call tests/test_simple.py::test_workers_health_custom_path[False]
5.09s call tests/test_simple.py::test_workers_health_custom_path[True]
5.08s call tests/test_simple.py::test_workers_health[True]
5.08s call tests/e2e/test_e2e.py::test_openai_embedding_parity
5.08s call tests/test_simple.py::test_workers_health[False]
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_tools
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_audio_input
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_image_input
5.06s call tests/e2e/test_e2e.py::test_openai_parity_with_response_format
5.03s call tests/e2e/test_e2e.py::test_e2e_default_spec
5.02s call tests/e2e/test_e2e.py::test_e2e_default_async_streaming
5.02s call tests/e2e/test_e2e.py::test_e2e_default_api
5.02s call tests/e2e/test_e2e.py::test_e2e_single_streaming
```


**Goal:**
Faster, more reliable CI without needing longer timeouts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the slow-test report and the analysis in PR #526, focusing on tests/test_simple.py::test_server_with_disabled_timeout and tests/test_readme.py::test_readme. Compare the other listed tests in tests/test_transport.py, tests/test_lit_server.py, tests/test_callbacks.py, and tests/e2e/test_e2e.py to identify expensive dependencies or waits. Done means the affected tests retain coverage and CI completes reliably without increased timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, performance, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.