mlcommons / mlcommons/inference
Interactive: still occasionally a separate workload rather than a fully fledged scenario?
Open
@pgmpablo157321 is already working on this.
Since Sep 30, 2025.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
It's confusing that mlperf.conf still uses separate workloads and the Server scenario for the Interactive case.
grep -Ri interactive inference/loadgen/*
docs/src/doxygen.cfg:# enable generation of interactive SVG images that allow zooming and panning. docs/src/doxygen.cfg:INTERACTIVE_SVG = YES mlperf.conf:llama2-70b-interactive.*.performance_sample_count_override = 24576 mlperf.conf:llama3_1-405b-interactive.*.performance_sample_count_override = 8313 mlperf.conf:llama3_1-8b-interactive.*.performance_sample_count_override = 13368 mlperf.conf:llama2-70b-interactive.*.sample_concatenate_permutation = 1 mlperf.conf:llama3_1-405b-interactive.*.sample_concatenate_permutation = 1 mlperf.conf:llama3_1-8b-interactive.*.sample_concatenate_permutation = 1 mlperf.conf:llama2-70b-interactive.*.use_token_latencies = 1 mlperf.conf:llama3_1-405b-interactive.*.use_token_latencies = 1 mlperf.conf:llama3_1-8b-interactive.*.use_token_latencies = 1 mlperf.conf:# Target Latencies for interactive setting mlperf.conf:llama2-70b-interactive.Server.target_latency = 0 mlperf.conf:llama2-70b-interactive.Server.ttft_latency = 450 mlperf.conf:llama2-70b-interactive.Server.tpot_latency = 40 mlperf.conf:# Target Latencies for interactive setting mlperf.conf:llama3_1-405b-interactive.Server.target_latency = 0 mlperf.conf:llama3_1-405b-interactive.Server.ttft_latency = 4500 mlperf.conf:llama3_1-405b-interactive.Server.tpot_latency = 80 mlperf.conf:# Target Latencies for interactive setting mlperf.conf:llama3_1-8b-interactive.Server.target_latency = 0 mlperf.conf:llama3_1-8b-interactive.Server.ttft_latency = 500 mlperf.conf:llama3_1-8b-interactive.Server.tpot_latency = 30
Still, I assume that user.conf for Interactive is expected to look like:
*.Server.target_qps = <target QPS lower than for Server to meet more stringent latency constraints>
*.Server.min_duration = <minimum duration in milliseconds; at least 600,000>
(* here should cover both llama2-70b-interactive and llama2-70b, whichever is correct.)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.