mlcommons / mlcommons/endpoints
bug: target_qps hardcoded to 10.0 in Offline mode instead of None
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Problem
In src/inference_endpoint/config/runtime_settings.py:142-148, target_qps falls back to a hardcoded 10.0 for Offline (max_throughput) mode instead of None:
# TODO: target_qps should be None in Offline mode but using 10.0 as fallback
# to avoid breaking changes
target_qps = config.settings.target_qps or 10.0
In Offline/max_throughput mode, target_qps is semantically irrelevant — all queries are issued at t=0 as a burst. Having it default to 10.0 is misleading and can affect any downstream logic that reads this field.
Expected Behavior
target_qps should be None when the load pattern is max_throughput. The fallback workaround should be removed and callers that depend on this field should be updated to handle None.
Files to Modify
src/inference_endpoint/config/runtime_settings.py- Any callers that read
runtime_settings.target_qpswithout a None check
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/inference_endpoint/config/runtime_settings.py:142-148 and trace how target_qps is assigned for max_throughput. Search for callers that read runtime_settings.target_qps, then update handling so Offline mode preserves None without breaking other load patterns. Done means target_qps is None for max_throughput and all affected callers handle it safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100