mlcommons/endpoints

View on GitHub

MLCommons Inference Endpoints repository

Stars: 21
Forks: 28
Open beginner issues: 0
Indexed issues: 69
Avg merge: 3d 17h
Merged PRs (30d): 13

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue

Beginner friendly Everything

Open All

69 open issues indexed

[Feature]: plug in steady state reporting Open

status: needs-triage type: feature

mlcommons/endpoints#510 · 1 assignee · Opened Sep 18, 2026
[Bug]: Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" during running performance(online, offline), sglang 0.5.15.post1, 0.5.18 Open

status: needs-triage type: bug

Difficulty 3/5 1-2 days Newbie friendliness 78/100

mlcommons/endpoints#497 · 1 reaction · Opened Sep 8, 2026
Add an overall deadline for the complete compliance audit sequence Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/endpoints#466 · Opened Aug 25, 2026
Parameterize run-timeout coverage across benchmark lifecycle stages Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

mlcommons/endpoints#465 · Opened Aug 25, 2026
Add a deadline for accuracy scoring and finalization Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

mlcommons/endpoints#464 · Opened Aug 25, 2026
[Feature]: populate `session_started_ns` correctly Open

status: needs-triage type: feature

Difficulty 3/5 1-2 days Newbie friendliness 68/100

mlcommons/endpoints#463 · 1 comment · Opened Aug 24, 2026
Consolidate run-outcome state flags into one abort/outcome model Open

Difficulty 5/5 Over a week Newbie friendliness 42/100

mlcommons/endpoints#459 · Opened Aug 20, 2026
schema.py: dedupe cyclopts help= vs Field(description=) across config fields Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

mlcommons/endpoints#451 · Opened Aug 18, 2026
Promote warmup to a first-class phase type with per-phase config Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

mlcommons/endpoints#449 · 1 comment · Opened Aug 14, 2026
Optimize structured chat-template tokenization drain at scale Open

mlcommons/endpoints#444 · 2 comments · 1 assignee · Opened Aug 13, 2026
[Feature]: swebench pyxis x86 support Open

area: evaluation priority: P1 type: enhancement type: feature

mlcommons/endpoints#440 · 1 assignee · Opened Aug 7, 2026
[Bug]: VLM (qwen3-235B) scoring fails: `inference_endpoint/evaluation/scoring.py` msgspec.DecodeError: JSON is malformed: invalid character (byte 145) Open

status: needs-triage type: bug

mlcommons/endpoints#427 · 2 comments · 1 assignee · Opened Jul 23, 2026
[Bug]: output config.yaml should respect --mode flag Open

status: needs-triage type: bug

Difficulty 3/5 1-2 days Newbie friendliness 62/100

mlcommons/endpoints#420 · Opened Jul 16, 2026
HTTP client: num_workers must be a multiple of the endpoint count for even load — auto-scale workers for multiple endpoints Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mlcommons/endpoints#419 · Opened Jul 16, 2026
Connection pool: treat EADDRNOTAVAIL as backpressure so the full-budget auto ceiling is safe under max_throughput bursts Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mlcommons/endpoints#418 · Opened Jul 16, 2026
Clarify BFCL / tool-call output token count (OSL is approximate for structured outputs) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/endpoints#410 · Opened Jul 14, 2026
compliance: scope edge-agentic checker (rename to edge_agentic.py, required ruleset/model args) Open

mlcommons/endpoints#397 · 1 assignee · Opened Jul 6, 2026
BFCL/edge: make single-stream a first-class HTTPClientConfig option (drop accuracy-only override) Open

mlcommons/endpoints#395 · 1 assignee · Opened Jul 6, 2026
BFCL/openai: harden msgspec adapter tests + hoist imports + gc docstring for tool_choice Open

mlcommons/endpoints#396 · 1 assignee · Opened Jul 6, 2026
BFCL scoring/reporting & robustness cleanup (live_relevance, no-metric PASS, drain cap, plot guard, dead code) Open

mlcommons/endpoints#393 · 1 assignee · Opened Jul 6, 2026
BFCL test coverage: BFCLv4MultiTurnScorer.score() and bfcl_v4 generate() selection logic Open

mlcommons/endpoints#394 · 1 assignee · Opened Jul 6, 2026
BFCL multi-turn: endpoint-controlled tool-call names reach bfcl-eval eval() (RCE on evaluator host) Open

mlcommons/endpoints#391 · 1 assignee · Opened Jul 6, 2026
BFCL multi-turn execution robustness: malformed 200 body + uncleaned bfcl-eval instance cache Open

mlcommons/endpoints#392 · 1 assignee · Opened Jul 6, 2026
Move bfcl-eval to an out-of-process uv subproject (remove parent-env pip caveats) Open

mlcommons/endpoints#390 · 1 assignee · Opened Jul 6, 2026
Schema-aware adapter selection (drop hard-required 'prompt' column / ColumnFilter skip) Open

mlcommons/endpoints#388 · 1 assignee · Opened Jul 1, 2026
Unify accuracy scoring path for perf+accuracy and accuracy-only runs Open

mlcommons/endpoints#387 · 1 assignee · Opened Jul 1, 2026
BFCL-v4 multi-turn runner: reuse HttpClientSampleIssuer instead of bespoke HTTP client Open

mlcommons/endpoints#382 · 1 reaction · 1 assignee · Opened Jun 30, 2026
[Bug]: Inconsistency with the field `min_duration_ms` in config.yaml Open

status: needs-triage type: bug

Difficulty 3/5 1-2 days Newbie friendliness 58/100

mlcommons/endpoints#373 · Opened Jun 23, 2026
OpenAI SSE accumulator assumes one token channel per streamed chunk Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mlcommons/endpoints#325 · Opened May 25, 2026
[Feature]: Audit Test (Test04 for Wan-2.2) Open

status: needs-triage type: feature

mlcommons/endpoints#320 · 1 comment · 4 assignees · Opened May 20, 2026
[Feature]: Inline accuracy check of performance runs Open

status: needs-triage type: feature

Difficulty 4/5 3-5 days Newbie friendliness 52/100

mlcommons/endpoints#313 · Opened May 15, 2026
[Feature]: cache salting for multi-turn Open

status: needs-triage type: feature

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/endpoints#312 · Opened May 15, 2026
[Feature]: Multiple dataset copies for multi-turn Open

status: needs-triage type: feature

mlcommons/endpoints#311 · 2 comments · 2 assignees · Opened May 15, 2026
Max connections limits load concurrency Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlcommons/endpoints#301 · Opened Apr 29, 2026
[Feature]: openai/gpt-oss-120b reference implementation Open

status: needs-triage type: feature

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/endpoints#298 · 3 comments · Opened Apr 28, 2026
[Feature]: Deepseek R1 reference implementation Open

status: needs-triage type: feature

mlcommons/endpoints#297 · 1 assignee · Opened Apr 28, 2026
[Discussion] Video Return Format and Transmission Overhead in Inference Performance Open

status: needs-triage type: feature

mlcommons/endpoints#289 · 1 assignee · Opened Apr 21, 2026
[Bug]: Custom dataset name not getting reflected in sample_idx_map.json Open

status: needs-triage type: bug

Difficulty 3/5 1-2 days Newbie friendliness 58/100

mlcommons/endpoints#284 · Opened Apr 16, 2026
Endpoints Storage Requirements - Add Accuracy requirements Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlcommons/endpoints#280 · 1 comment · Opened Apr 13, 2026
Tokenizer boundary impact Open

Difficulty 5/5 Over a week Newbie friendliness 32/100

mlcommons/endpoints#278 · 1 comment · Opened Apr 10, 2026
Low concurrency timeouts Open

area: client priority: P1 type: bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/endpoints#269 · Opened Apr 3, 2026
feat: multi-turn implementation Open

area: dataset priority: P0 type: feature

mlcommons/endpoints#232 · 1 comment · 2 assignees · Opened Mar 31, 2026
Endpoints test environment Open

priority: P2 type: chore

mlcommons/endpoints#229 · 1 comment · 1 assignee · Opened Mar 30, 2026
Add additional DB and Object Store elements from Requirements Doc Open

priority: P2 type: feature

mlcommons/endpoints#227 · 1 assignee · Opened Mar 30, 2026
feat: multiple performance configs in a submission config Open

area: config-cli priority: P2 type: feature

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mlcommons/endpoints#224 · 3 comments · Opened Mar 27, 2026
[Roadmap] Inference Endpoints Phase 2 Roadmap Open

type: RFC

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/endpoints#223 · Opened Mar 27, 2026
test: KVStore, ServiceLauncher, and MetricsAggregator services lack integration tests Open

area: core-engine priority: P1 type: chore

Difficulty 4/5 3-5 days Newbie friendliness 48/100

mlcommons/endpoints#222 · Opened Mar 27, 2026
bug: RuntimeSettings stores Random objects making benchmarks non-reproducible and non-serializable Open

area: config-cli priority: P1 type: bug

Difficulty 3/5 1-2 days Newbie friendliness 67/100

mlcommons/endpoints#221 · Opened Mar 27, 2026
test: SGLang adapter integration tests are permanently skipped Open

area: adapters priority: P1 type: chore

Difficulty 3/5 1-2 days Newbie friendliness 55/100

mlcommons/endpoints#220 · Opened Mar 27, 2026
bug: target_qps hardcoded to 10.0 in Offline mode instead of None Open

area: config-cli priority: P1 type: bug

Difficulty 3/5 1-2 days Newbie friendliness 65/100

mlcommons/endpoints#219 · 1 comment · Opened Mar 27, 2026

mlcommons/endpoints

Get new issues in your inbox