InseeFrLab / InseeFrLab/auto-tuning-vllm
Expose GuideLLM benchmark worker/process count in autotuner configuration
- Dominant language
- Python
- Stars
- 6
- Forks
- 0
- Avg merge
- 34m
- Merged PRs (30d)
- 2
Description
When running GuideLLM benchmarks in the same Kubernetes pod as the vLLM server, the benchmark client can saturate the available CPU resources under high concurrency workloads.
In the current autotuner implementation, the number of GuideLLM benchmark processes/workers is not configurable, even though this parameter is supported by GuideLLM itself.
This can lead to situations where benchmark-side CPU usage impacts the measured metrics when benchmark and serving workloads share the same pod.
Proposed improvement:
* expose the GuideLLM process/worker count as a configurable parameter in the autotuner.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the autotuner configuration and the code that launches GuideLLM benchmarks. Expose the GuideLLM process/worker count there, then verify that the configured value is passed to benchmark runs and limits benchmark-side CPU usage under concurrent workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100