[Roadmap] Inference Endpoints Phase 2 Roadmap

We will document the issues that are required to reach key milestones in this document.

How to join us

• Subscribe to our mailing list: link
• Join our Slack: link

Goal

Post-GTC / Q2 2026: Stabilize the benchmarking platform, expand accuracy coverage, harden reliability, and close all known P0 gaps following the GTC demo.

See Phase 1 roadmap: #83

---

ShowStopper

• Accuracy evaluation for LLMs (GPQA, AIME, HLE, etc.) #4
• Throughput/latency parity with MLPerf LoadGen and vLLM benchmark_serving #8
• Pareto submission clarification (models, datasets, points) #84

---

Functionality (P0)

• Submission ruleset interaction with config (Phase 4 integration) #5
• Define and implement the final report structure #6
• Analyze roofline of inference endpoints (offline/online/SSE) #9
• Pub/Sub implementation for event recorder #183
• Remove unsafe serialization support (security: arbitrary code execution risk) #184
• EVAL and SUBMISSION test types raise bare NotImplementedError — add user-friendly errors #218

---

Bug Fixes

• PostgreSQL DuplicatePreparedStatement error in recorder #213
• max_throughput mode causes connection timeouts #202
• Performance discrepancy between submission config and performance-only config #199
• max_duration_ms ends online test prematurely before all samples are issued #197
• target_qps hardcoded to 10.0 in Offline mode instead of None #219
• RuntimeSettings stores Random objects — benchmarks non-reproducible and non-serializable #221
• Errors seen running endpoint tests in local mode #99
• llama.cpp incompatibility (requires OpenAI-compatible server) #133

---

Accuracy & Datasets (P1)

• MMLU / MMLU-Pro — dataset + scorer + extractor #176
• MATH500 — dataset integration #177
• Healthbench integration #178
• Humanity's Last Exam (HLE) integration #179

---

Performance & Benchmarking (P1)

• System bottleneck tests (client stress tests) #10
• Explain text vs token performance regression on TRTLLM #182
• Implement BURST and STEP load patterns #217

---

Testing & CI (P0/P1)

• Add GitLab CI to stress test upper bound of endpoint measurement #138
• SGLang adapter integration tests are permanently skipped #220
• KVStore, ServiceLauncher, and MetricsAggregator services lack integration tests #222

---

UI/UX & Visualization (P1)

• Runtime visualization of the benchmark #7

Start by reviewing the roadmap checklist and the linked issues for the milestone areas that still need work. This issue is complete when the required Phase 2 work is tracked and the roadmap accurately reflects the status of its linked issues.