mlcommons / mlcommons/endpoints
feat: multi-turn implementation
Open
@hvagadia is already working on this.
Since Jul 22, 2026.
area: dataset
priority: P0
type: feature
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Working PR:
https://github.com/mlcommons/endpoints/pull/226
Motivation and Use Cases
Why Multi-Turn?
Modern conversational AI systems like ChatGPT, Claude, and production chatbots operate in multi-turn mode where:
- Context matters: Each response depends on conversation history
- Sequential dependencies: Turn N+1 must wait for turn N's response
- Realistic workloads: Single-turn benchmarks don't reflect production patterns
- Performance characteristics differ: Multi-turn introduces blocking, queueing, and memory accumulation
High-Level Architecture
Component Diagram
┌─────────────────────────────────────────────────────────────────┐
│ Load Generator │
│ │
│ ┌───────────────────┐ ┌──────────────────┐ │
│ │ MultiTurnDataset │────▶│ ConversationMgr │ │
│ │ (JSONL loader) │ │ (State tracking)│ │
│ └───────────────────┘ └──────────────────┘ │
│ │ │ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ MultiTurnScheduler (Turn Sequencing) │ │
│ │ • Parallel mode (all turn-1s at t=0) │ │
│ │ • Sequential mode (complete conv by conv) │ │
│ │ • Poisson mode (arrival distribution) │ │
│ │ • Optional concurrency control │ │
│ └──────────────────────────────────────────────┘ │
│ │ │
└───────────┼──────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Endpoint Client │
│ Worker Process → HTTP Request → OpenAI Adapter │
│ (Forwards conversation_id + turn_number in metadata) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Metrics Recorder │
│ EventRecorder: SQLite events.db with conversation fields │
│ MetricsReporter: Per-turn and per-conversation metrics │
└─────────────────────────────────────────────────────────────────┘
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.