InseeFrLab / InseeFrLab/auto-tuning-vllm

Expose GuideLLM benchmark worker/process count in autotuner configuration

Open Beginner friendly
#26 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
0
Avg merge
34m
Merged PRs (30d)
2

Description

When running GuideLLM benchmarks in the same Kubernetes pod as the vLLM server, the benchmark client can saturate the available CPU resources under high concurrency workloads.

In the current autotuner implementation, the number of GuideLLM benchmark processes/workers is not configurable, even though this parameter is supported by GuideLLM itself.

This can lead to situations where benchmark-side CPU usage impacts the measured metrics when benchmark and serving workloads share the same pod.

Proposed improvement:

* expose the GuideLLM process/worker count as a configurable parameter in the autotuner.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the autotuner configuration and the code that launches GuideLLM benchmarks. Expose the GuideLLM process/worker count there, then verify that the configured value is passed to benchmark runs and limits benchmark-side CPU usage under concurrent workloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
2/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.