mlcommons / mlcommons/modelbench
test in HOWTO fails
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
Following the instructions here:
https://github.com/mlcommons/modelbench/blob/main/README.md
I get a TOML error:
knoe:modelbench kurt$ poetry run modelbench benchmark -m 10
Version 0.5 of this benchmark is a proof of concept only. Results are not intended to indicate actual levels of AI system safety.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 100, in benchmark
benchmark_scores = score_benchmarks([benchmark], suts, max_instances, json_logs, debug)
File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 129, in score_benchmarks
run = run_benchmarks_for_suts(benchmarks, suts, max_instances, debug=debug, json_logs=json_logs)
File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 140, in run_benchmarks_for_suts
runner.secrets = load_secrets_from_config()
File "/Users/kurt/git/modelbench/src/modelgauge/config.py", line 30, in load_secrets_from_config
data = tomli.load(f)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 66, in load
return loads(s, parse_float=parse_float)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 102, in loads
pos = key_value_rule(src, pos, out, header, parse_float)
File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 349, in key_value_rule
raise suffixed_err(src, pos, "Cannot overwrite a value")
tomli.TOMLDecodeError: Cannot overwrite a value (at line 11, column 77)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the setup and benchmark instructions in README.md, then inspect src/modelgauge/config.py, especially load_secrets_from_config, while reproducing poetry run modelbench benchmark -m 10. Determine which documented TOML configuration causes the duplicate-value error at line 11, update the relevant instructions or configuration handling, and verify that the command runs without the TOMLDecodeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, documentation
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100