microsoft / microsoft/superbenchmark

Config file auto-generation

Open
#707 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
393
Forks
90
PR merge metrics
No merged PRs in 30d

Description

What would you like to be added:

Config file auto-generation feature.

Why is this needed:

Creating a configuration file from scratch is overly complicated. Users must manually select valid benchmarks and configure the appropriate parameters for each test, which increases the likelihood of errors (e.g., misspellings).

Without this feature, how does current superbenchmark work

Users need to manually write superbench config YAML file (and/or rule files) to run benchmarks, diagnosis, and result summary.

Components that may involve changes:

cli

Brief description of your proposal if any:

Can we have a feature similar to nhc's config file auto-generation, which can automatically generate an initial config file based on the node info scan?

It can scan the basic node info and generate an initial sample config by:

  1. Selecting all the valid benchmarks for the node.
  2. Setting parameters for each benchmark test (e.g., set num_proc based on the number of GPUs).

And then the user can easily modify the config for some customization (e.g., remove some unnecessary benchmarks and adjust parameters).

It can be a new subcommand (e.g., sb config generate) in the CLI, or just some utility scripts.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CLI component and trace how node information is scanned and how benchmark configuration YAML or rule files are currently handled. Define the scope of a config-generation entry point such as sb config generate; done means it produces an initial config containing valid benchmarks and suitable parameters that users can customize.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.