deepspeedai / deepspeedai/DeepSpeedExamples
Why does the rw_eval.py script return a different score for the same sample each time?
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
- model: llama-2-7b-hf
- execute command: python rw_eval.py --model_name_or_path /data/llama-2-hf/llama-2-7b-hf/
- GPU: A6000(48G)
- result
-
first result
-
second result
- Question: Why does the rw_eval.py script return a different score for the same sample each time? How to modify to make the same sample to return the same score each time?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rw_eval.py and reproduce the issue using the provided python command, model path, and llama-2-7b-hf model. Trace the evaluation path for sources of run-to-run variation, then verify that repeated evaluations of the same sample produce the same score after the change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100