mlcommons / mlcommons/endpoints
[Bug]: output config.yaml should respect --mode flag
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Bug Description
When --mode is specified as performance (or accuracy), if the source config file contains both the performance and accuracy dataset, the output should strip the dataset not used to ensure that the config file reproduces the run.
Steps to Reproduce
PR #342 examples/10_Agentic_Inference/accuracy/RUNBOOK.md contains an example of a config with both modes.
Environment
Config file with both perf/accuracy dataset.
Relevant Logs
Before submitting
- I searched existing issues and found no duplicates
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the --mode handling and the output-config generation path, using the example in examples/10_Agentic_Inference/accuracy/RUNBOOK.md as the reproduction reference. Verify behavior for both performance and accuracy modes when both datasets are present. Done means the emitted config retains only the dataset selected by --mode and reproduces the run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100