mlfoundations / mlfoundations/evalchemy
How to change model's temperature ?
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 610
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I could not find an obvious way from the given examples on how one could change the model temparature (and number of runs, which defaults to 5) used for the generations?
For example, if i am running the script like this
python -m eval.eval\ --model vllm \ --tasks AIME24,AIME25,MATH500,GPQADiamond,AMC23 \ --model_args "pretrained=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" \ --batch_size 4 \ --output_path logs
It looks like the temperature of 0.7 is used (but with do_sample = False for some reason) how can I change this? passing temperature to model_args doesn't recognize it
Thanks a lot!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the eval.eval command entry point and trace how model_args, temperature, do_sample, and the default number of runs are handled. Confirm the currently supported invocation and document how to set these values, or clarify the missing configuration behavior; done when the example command and resulting settings are unambiguous.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100