google-deepmind / google-deepmind/gemma

how about ppl on wikitext?

Open
#44 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

Hi, i read the Gemma technical reports, which show many evaluation results, but i did not find any results about PPL. then i tried to evaluate Gemma PPL, but got not good results(much worse than some other mainstream models, eg.llama2 ) , as follows:

gemma-2b ppl=13.8754587(seqlen=2048)
gemma-2-2b ppl= 68.940315(2048), 49.55793(8192)
gemma-7b= 91195840.0(2048)
gemma-2-9b ppl=198.7406(2048),65.0805(8192)

can u provide PPL on wikitext benachmark? and how to evaluate PPL to get comparable results?

besides, i can get comparable mmlu results(reported in huggingface) bby LM-EVAL

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.