mlcommons / mlcommons/endpoints

feat: Implement BURST and STEP load patterns

Open
#217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: core-engine priority: P2 type: feature
Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Problem

LoadPatternType.BURST and LoadPatternType.STEP are defined in the config schema enum but have no scheduler implementation:

# src/inference_endpoint/config/schema.py:72-73
BURST = "burst"
STEP = "step"

A user who selects either pattern in their YAML config will get no error, but also no expected behavior — the scheduler will silently fall through or raise an unhandled case.

Expected Behavior

  • BURST: Issue a configurable burst of N queries at a fixed interval (e.g. N queries every T seconds)
  • STEP: Incrementally step up QPS in stages (useful for finding saturation point)

Files to Modify

  • src/inference_endpoint/config/schema.py — document the pattern parameters
  • src/inference_endpoint/load_generator/scheduler.py — add scheduler implementations
  • src/inference_endpoint/config/runtime_settings.py — ensure RuntimeSettings handles these patterns

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LoadPatternType entries at src/inference_endpoint/config/schema.py:72-73, then inspect src/inference_endpoint/load_generator/scheduler.py and src/inference_endpoint/config/runtime_settings.py to understand existing pattern handling and configuration conventions. Done means BURST and STEP configurations are accepted and produce the documented query schedules, with coverage for both patterns and their parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.