huggingface / huggingface/lighteval
issues when testing wikitext
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Hi, I'm using the lighteval to test several benchmarks, but I met issues when testing the following 2 benchmarks:
1. lighteval|wikitext|0|0
2. helm|wikitext:103|0|0
When test wikitext, I got error
`Traceback (most recent call last):
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/run_evals_accelerate.py", line 97, in
main(args)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/logging/hierarchical_logger.py", line 144, in wrapper
return fn(*args, **kwargs)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/main_accelerate.py", line 71, in main
requests, docs = create_requests_from_tasks(
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 615, in create_requests_from_tasks
reqs = task.construct_requests(doc, ctx, doc_id_seed, cur_task_name)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 380, in construct_requests
LoglikelihoodRollingRequest(task_name=current_task_name, doc_id=document_id_seed, ctx=context)#LoglikelihoodRollingRequest(task_name=current_task_name, example_index=document_id_seed, request_index=0, context=context)
TypeError: LoglikelihoodRollingRequest.__init__() got an unexpected keyword argument 'doc_id'
`
I modified https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/lighteval_task.py#L380 to `LoglikelihoodRollingRequest(task_name=current_task_name, example_index=document_id_seed, request_index=0, context=context)` ends up with Nan perplexity.
When run wikitext:103, I got error like
`Traceback (most recent call last):
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/run_evals_accelerate.py", line 97, in
main(args)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/logging/hierarchical_logger.py", line 144, in wrapper
return fn(*args, **kwargs)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/main_accelerate.py", line 71, in main
requests, docs = create_requests_from_tasks(
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 563, in create_requests_from_tasks
task_dict_items = [(name, task) for name, task in task_dict.items() if len(task.eval_docs()) > 0]
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 563, in
task_dict_items = [(name, task) for name, task in task_dict.items() if len(task.eval_docs()) > 0]
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 296, in eval_docs
self._docs = self._get_docs_from_split(self.evaluation_split)
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/lighteval_task.py", line 266, in _get_docs_from_split
docs.extend(as_list(self.formatter(item, self.name)))
File "/group/ossdphi_algo_scratch_04/fuweiy/LLM/eval_v100/lighteval/src/lighteval/tasks/tasks_prompt_formatting.py", line 2068, in wikitext_103
return Doc(task_name=task_name, query=line["text"])
TypeError: Doc.__init__() missing 2 required positional arguments: 'choices' and 'gold_index'`
Commands I used are
`python run_evals_accelerate.py --model_args="pretrained=gpt2" --tasks "lighteval|wikitext|0|0" --override_batch_size 1 --save_details --output_dir="tmp/"` and `python run_evals_accelerate.py --model_args="pretrained=gpt2" --tasks "helm|wikitext:103|0|0" --override_batch_size 1 --save_details --output_dir="tmp/"`
Any advice on the issues? Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.