google-deepmind / google-deepmind/gemma
how about ppl on wikitext?
Open
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1k
- Avg merge
- 10h 33m
- Merged PRs (30d)
- 2
Description
Hi, i read the Gemma technical reports, which show many evaluation results, but i did not find any results about PPL. then i tried to evaluate Gemma PPL, but got not good results(much worse than some other mainstream models, eg.llama2 ) , as follows:
gemma-2b ppl=13.8754587(seqlen=2048)
gemma-2-2b ppl= 68.940315(2048), 49.55793(8192)
gemma-7b= 91195840.0(2048)
gemma-2-9b ppl=198.7406(2048),65.0805(8192)
can u provide PPL on wikitext benachmark? and how to evaluate PPL to get comparable results?
besides, i can get comparable mmlu results(reported in huggingface) bby LM-EVAL
Contributor guide
Assessment
This issue has not been assessed yet.