mlcommons / mlcommons/modelbench
Improve how locales and prompt file name interact with each other
Open
@rogthefrog is already working on this.
Since Jan 9, 2025.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
The safe test downloads prompt files from modellab. It supports multiple locales in a given file, but doesn't do well with different files for different locales.
Workarounds exist, but they're hacky.
Options:
- Improve file naming conventions to support locales. Proposal: if a file includes one locale, include that locale in the filename. English prompt files don't include en_us and that makes the code messy.
- Ensure that we don't try to make test items in locale X using a file that does not include locale X (e.g. in the smoke test).
- Handle file not found errors better, especially in unit or smoke tests.
- If any unit tests rely on downloading an asset from modellab, replace that with a local file.
References:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.