vllm-project/guidellm

View on GitHub

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Stars: 1.6k
Forks: 234
Open beginner issues: 0
Indexed issues: 50
Avg merge: 2d 22h
Merged PRs (30d): 46

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 20, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

50 open issues indexed

Native Multi-LoRA / Multi-Tenant Workload Simulation and Weighted Targeting Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

vllm-project/guidellm#1158 · Opened Sep 17, 2026
Discuss bilingual (English/Chinese) documentation support Open

Difficulty 5/5 Over a week Newbie friendliness 45/100

vllm-project/guidellm#1156 · 1 comment · Opened Sep 17, 2026
Ensure requests are forcefully killed a certain amount of time after benchmark completes. Open

internal invalid needs followup

Difficulty 3/5 1-2 days Newbie friendliness 45/100

vllm-project/guidellm#1152 · Opened Sep 16, 2026
Fix dropped modalities in client, when passed synthetic_text data kind Open

vllm-project/guidellm#1147 · 1 assignee · Opened Sep 14, 2026
OMNI TTS Open

feature invalid

vllm-project/guidellm#1125 · 1 assignee · Opened Sep 10, 2026
vLLM OMNI support Open

feature invalid

vllm-project/guidellm#1124 · 1 assignee · Opened Sep 10, 2026
Support inline and file-based API keys with global per-request rotation Open

Difficulty 4/5 3-5 days Newbie friendliness 50/100

vllm-project/guidellm#1104 · Opened Sep 8, 2026
Add conversation-level concurrency for realistic multi-turn user and agent workloads Open

vllm-project/guidellm#1082 · 1 comment · 1 assignee · Opened Sep 3, 2026
Enhancement: Persistent dataset generation with parallel workers and deterministic seeding Open

vllm-project/guidellm#1033 · 3 comments · 1 assignee · Opened Aug 19, 2026
Support loading pre-generated multi-turn conversation datasets from file Open

vllm-project/guidellm#1024 · 2 comments · 1 assignee · Opened Aug 13, 2026
Add OpenTelemetry tracing and context propagation Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

vllm-project/guidellm#986 · Opened Aug 3, 2026
[Bug]: Multipart media payloads are Base64-embedded in benchmark JSON and HTML reports Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

vllm-project/guidellm#984 · Opened Aug 3, 2026
Implement OTEL trace format replay Open

feature internal priority-medium

Difficulty 4/5 3-5 days Newbie friendliness 45/100

vllm-project/guidellm#968 · 3 comments · Opened Jul 27, 2026
Agentic trace replay capabilities Open

feature internal priority-high

Difficulty 5/5 Over a week Newbie friendliness 45/100

vllm-project/guidellm#965 · Opened Jul 27, 2026
Feature Request: Compare metrics across different benchmarks support Open

Difficulty 5/5 Over a week Newbie friendliness 45/100

vllm-project/guidellm#924 · Opened Jul 10, 2026
Configurable backend RTF for realtime transcription Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

vllm-project/guidellm#834 · Opened Jun 23, 2026
Audio transcription metrics (RTF & RTFx) Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

vllm-project/guidellm#833 · Opened Jun 23, 2026
Turn and Conversation Metrics Plan Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

vllm-project/guidellm#722 · 1 comment · Opened May 7, 2026
Throughput statistics for min/max/stdev/percentiles do not give a good representation of the benchmark Open

community contribution priority-low

vllm-project/guidellm#602 · 4 comments · 1 assignee · Opened Feb 19, 2026
Docs: End to end use case examples expansion / top workload pathways Open

documentation

Difficulty 4/5 3-5 days Newbie friendliness 35/100

vllm-project/guidellm#585 · 1 comment · Opened Feb 3, 2026
Alerting on internal scheduling problems Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

vllm-project/guidellm#574 · 1 comment · 1 reaction · Opened Feb 2, 2026
Investigate logging efficiency – package choice, style, etc. Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

vllm-project/guidellm#572 · Opened Feb 2, 2026
Support exporting live responses from ABI entrypoint Open

priority-medium

Difficulty 5/5 Over a week Newbie friendliness 25/100

vllm-project/guidellm#566 · Opened Jan 28, 2026
Support multimodal synthetic data Open

internal priority-medium

Difficulty 5/5 Over a week Newbie friendliness 25/100

vllm-project/guidellm#565 · 2 comments · Opened Jan 28, 2026
Improve throughput upper limit detection Open

internal priority-medium

Difficulty 5/5 Over a week Newbie friendliness 35/100

vllm-project/guidellm#564 · 1 reaction · Opened Jan 28, 2026
Define standardized scenarios Open

internal priority-low

Difficulty 4/5 3-5 days Newbie friendliness 35/100

vllm-project/guidellm#563 · Opened Jan 28, 2026
Documentation and Extension Points for Custom Backends Open

priority-low

Difficulty 5/5 Over a week Newbie friendliness 30/100

vllm-project/guidellm#551 · 3 comments · Opened Jan 23, 2026
Add support for vLLMs new gRPC interface Open

priority-medium

vllm-project/guidellm#548 · 1 comment · 1 assignee · Opened Jan 22, 2026
Get progress metrics in a file during run, not after the result is completed Open

needs followup

Difficulty 4/5 3-5 days Newbie friendliness 48/100

vllm-project/guidellm#518 · 4 comments · Opened Dec 15, 2025
Can you list some popular HuggingFace datasets that we can used to set guidellm --data Open

documentation

Difficulty 4/5 3-5 days Newbie friendliness 40/100

vllm-project/guidellm#505 · 3 reactions · Opened Dec 6, 2025
Support direct collection of vLLM metrics from Prometheus `/metrics` endpoint Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

vllm-project/guidellm#457 · 7 comments · 1 reaction · Opened Nov 11, 2025
Define workload characteristics for different application/enterprise use cases Open

documentation

Difficulty 5/5 Over a week Newbie friendliness 35/100

vllm-project/guidellm#437 · 2 comments · 2 reactions · Opened Oct 28, 2025
Sample prompts not displaying correctly Open

needs followup UI

Difficulty 3/5 1-2 days Newbie friendliness 48/100

vllm-project/guidellm#429 · 1 comment · Opened Oct 21, 2025
Better tracking of internal performance of load testing and reporting impacts/potential impacts to performance measurements and results Open

internal

Difficulty 5/5 Over a week Newbie friendliness 20/100

vllm-project/guidellm#385 · 1 reaction · Opened Oct 1, 2025
[Refactor] guidellm profiling enablement with pyspy to determine process slow downs Open

internal priority-medium

vllm-project/guidellm#346 · 2 assignees · Opened Sep 18, 2025
Create a contrib folder for user submitted and maintained guidellm output visualization Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

vllm-project/guidellm#304 · 1 comment · Opened Sep 5, 2025
Margin Of Error (MOE) stopping Open

internal

vllm-project/guidellm#244 · 1 assignee · Opened Jul 27, 2025
Margin Of Errors (MOE) in output report Open

internal

vllm-project/guidellm#243 · 1 comment · 1 assignee · Opened Jul 27, 2025
[Feature Request] Need for ability to save partial results Open

internal

Difficulty 4/5 3-5 days Newbie friendliness 45/100

vllm-project/guidellm#227 · 1 comment · Opened Jul 11, 2025
Bug: Negative TTFT values when benchmarking Ollama at high concurrency Open

internal

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

vllm-project/guidellm#216 · 4 comments · 1 reaction · Opened Jul 1, 2025
Improve error handling around invalid tokenizer names Open

cli internal needs followup

Difficulty 3/5 1-2 days Newbie friendliness 52/100

vllm-project/guidellm#205 · 5 comments · Opened Jun 26, 2025
Token Efficiency Benchmarking Open

internal

Difficulty 5/5 Over a week Newbie friendliness 35/100

vllm-project/guidellm#179 · 3 comments · Opened May 31, 2025
[Feature Request] Record server config from /server_info endpoint when vLLM 0.9.0 lands Open

internal

vllm-project/guidellm#166 · 1 assignee · Opened May 19, 2025
[Feature Request] TTFT & ITL Timeouts Open

internal priority-low

Difficulty 4/5 3-5 days Newbie friendliness 45/100

vllm-project/guidellm#107 · 1 comment · 1 reaction · Opened Apr 8, 2025
Enable Batch Inferencing Benchmarking Support Open

internal priority-medium

Difficulty 4/5 3-5 days Newbie friendliness 45/100

vllm-project/guidellm#102 · Opened Mar 31, 2025
Define effective hierarchy of debug logging Open

internal priority-low

vllm-project/guidellm#97 · 1 reaction · 2 assignees · Opened Mar 11, 2025
Have GuideLLM kick off a vLLM server automatically to avoid having the user install vLLM and assign the target themselves Open

internal

vllm-project/guidellm#95 · 1 comment · 2 assignees · Opened Mar 9, 2025
Max RPS in sweep mode could be higher Open

internal

Difficulty 3/5 1-2 days Newbie friendliness 45/100

vllm-project/guidellm#93 · 7 comments · Opened Mar 7, 2025
Enable vLLM scheduler specific Metrics Open

internal

vllm-project/guidellm#86 · 2 assignees · Opened Mar 4, 2025
[Scoping] Enable Hardware Accelerator Monitoring metrics to be captured in benchmarks Open

internal

Difficulty 5/5 Over a week Newbie friendliness 25/100

vllm-project/guidellm#85 · 1 reaction · Opened Mar 3, 2025

vllm-project/guidellm

Get new issues in your inbox