mlcommons / mlcommons/modelbench

test in HOWTO fails

Open
#558 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
134
Forks
36
Avg merge
1d 11h
Merged PRs (30d)
17

Description

Following the instructions here:

https://github.com/mlcommons/modelbench/blob/main/README.md

I get a TOML error:

knoe:modelbench kurt$ poetry run modelbench benchmark -m 10

Version 0.5 of this benchmark is a proof of concept only. Results are not intended to indicate actual levels of AI system safety.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 100, in benchmark
    benchmark_scores = score_benchmarks([benchmark], suts, max_instances, json_logs, debug)
  File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 129, in score_benchmarks
    run = run_benchmarks_for_suts(benchmarks, suts, max_instances, debug=debug, json_logs=json_logs)
  File "/Users/kurt/git/modelbench/src/modelbench/run.py", line 140, in run_benchmarks_for_suts
    runner.secrets = load_secrets_from_config()
  File "/Users/kurt/git/modelbench/src/modelgauge/config.py", line 30, in load_secrets_from_config
    data = tomli.load(f)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 66, in load
    return loads(s, parse_float=parse_float)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 102, in loads
    pos = key_value_rule(src, pos, out, header, parse_float)
  File "/Users/kurt/Library/Caches/pypoetry/virtualenvs/modelbench-hvJ58ebH-py3.10/lib/python3.10/site-packages/tomli/_parser.py", line 349, in key_value_rule
    raise suffixed_err(src, pos, "Cannot overwrite a value")
tomli.TOMLDecodeError: Cannot overwrite a value (at line 11, column 77)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the setup and benchmark instructions in README.md, then inspect src/modelgauge/config.py, especially load_secrets_from_config, while reproducing poetry run modelbench benchmark -m 10. Determine which documented TOML configuration causes the duplicate-value error at line 11, update the relevant instructions or configuration handling, and verify that the command runs without the TOMLDecodeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, documentation
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.