mlfoundations / mlfoundations/evalchemy
lm-evaluation-harness dependency conflicts
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 610
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
After pip install using the current TOML, and running the following command --
python -m eval.eval \
--model hf \
--tasks "leaderboard" \
--model_args \"pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16\" \
--batch_size auto \
--output_path logs
The run fails (after successfully completing all evaluation datapoints) because of a dependency conflict in the LaTeX parser.
2024-12-21:00:47:07,159 ERROR [utils.py:180] LaTeX parsing requires the antlr4 Python package, provided by pip (antlr4-python3-runtime) or conda (antlr-python-runtime), version 4.11
2024-12-21:00:47:07,160 ERROR [eval.py:213] Error in pretrain evaluation: LaTeX parsing requires the antlr4 Python package, provided by pip (antlr4-python3-runtime) or conda (antlr-python-runtime), version 4.11
Upgrading 'antlr4-python3-runtime' introduces a new dependency conflict --
omegaconf 2.2.3 requires antlr4-python3-runtime==4.9.*, but you have antlr4-python3-runtime 4.11.0 which is incompatible.
Upon upgrading omegaconf, another new dependency conflict appears --
sagemaker 2.237.1 requires omegaconf<2.3,>=2.2, but you have omegaconf 2.3.0 which is incompatible.
SUGGESTION: make Cloud & Storage dependencies optional.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the current TOML dependency definitions and reproduce the failure using the provided python -m eval.eval command. Trace the antlr4-python3-runtime, omegaconf, and sagemaker constraints; done means the evaluation runs without conflicts while Cloud and Storage dependencies can remain optional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, latex, python
- Domain
- build-system, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100