microsoft / microsoft/superbenchmark
Config file auto-generation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 393
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
What would you like to be added:
Config file auto-generation feature.
Why is this needed:
Creating a configuration file from scratch is overly complicated. Users must manually select valid benchmarks and configure the appropriate parameters for each test, which increases the likelihood of errors (e.g., misspellings).
Without this feature, how does current superbenchmark work:
Users need to manually write superbench config YAML file (and/or rule files) to run benchmarks, diagnosis, and result summary.
Components that may involve changes:
cli
Brief description of your proposal if any:
Can we have a feature similar to nhc's config file auto-generation, which can automatically generate an initial config file based on the node info scan?
It can scan the basic node info and generate an initial sample config by:
- Selecting all the valid benchmarks for the node.
- Setting parameters for each benchmark test (e.g., set
num_procbased on the number of GPUs).
And then the user can easily modify the config for some customization (e.g., remove some unnecessary benchmarks and adjust parameters).
It can be a new subcommand (e.g., sb config generate) in the CLI, or just some utility scripts.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the CLI component and trace how node information is scanned and how benchmark configuration YAML or rule files are currently handled. Define the scope of a config-generation entry point such as sb config generate; done means it produces an initial config containing valid benchmarks and suitable parameters that users can customize.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100