mlcommons / mlcommons/endpoints

feat: multi-turn implementation

Open
#232 1 comment 0 reactions 2 assignees View on GitHub

@hvagadia is already working on this.

Since Jul 22, 2026.

area: dataset priority: P0 type: feature
Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Working PR:
https://github.com/mlcommons/endpoints/pull/226

Motivation and Use Cases

Why Multi-Turn?

Modern conversational AI systems like ChatGPT, Claude, and production chatbots operate in multi-turn mode where:

  1. Context matters: Each response depends on conversation history
  2. Sequential dependencies: Turn N+1 must wait for turn N's response
  3. Realistic workloads: Single-turn benchmarks don't reflect production patterns
  4. Performance characteristics differ: Multi-turn introduces blocking, queueing, and memory accumulation

High-Level Architecture

Component Diagram
┌─────────────────────────────────────────────────────────────────┐
│                        Load Generator                            │
│                                                                   │
│  ┌───────────────────┐     ┌──────────────────┐                │
│  │ MultiTurnDataset  │────▶│ ConversationMgr  │                │
│  │  (JSONL loader)   │     │  (State tracking)│                │
│  └───────────────────┘     └──────────────────┘                │
│           │                          │                           │
│           │                          │                           │
│           ▼                          ▼                           │
│  ┌──────────────────────────────────────────────┐              │
│  │      MultiTurnScheduler (Turn Sequencing)     │              │
│  │   • Parallel mode (all turn-1s at t=0)       │              │
│  │   • Sequential mode (complete conv by conv)   │              │
│  │   • Poisson mode (arrival distribution)       │              │
│  │   • Optional concurrency control              │              │
│  └──────────────────────────────────────────────┘              │
│           │                                                      │
└───────────┼──────────────────────────────────────────────────────┘
            │
            ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Endpoint Client                             │
│  Worker Process → HTTP Request → OpenAI Adapter                 │
│  (Forwards conversation_id + turn_number in metadata)           │
└─────────────────────────────────────────────────────────────────┘
            │
            ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Metrics Recorder                            │
│  EventRecorder: SQLite events.db with conversation fields        │
│  MetricsReporter: Per-turn and per-conversation metrics          │
└─────────────────────────────────────────────────────────────────┘

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.