huggingface / huggingface/lighteval

[FT] pass trust_remote_code as flag for loading datasets with custom code

Open
#314 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
When trying to run an evaluation while loading a dataset with custom code, there doesn't seem to be an easy way to set the option `trust_remote_code=True`.

The command line will say this:
```
The repository for swiss_leading_decision_summarization_eval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/swiss_leading_decision_summarization_eval.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N]
```

If the user can't type in `y` quick enough (not within 10 ish seconds), there is an error:
```
File "/Users/chuandu/Documents/workspace/legal_llm_evaluation/llm_eval_env/lib/python3.10/site-packages/datasets/load.py", line 131, in resolve_trust_remote_code
raise ValueError(
ValueError: The repository for swiss_leading_decision_summarization_eval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/swiss_leading_decision_summarization_eval.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
```

## Solution/Feature
Currently we can pass `trust_remote_code=True` for the model, but a separate variable is needed for loading datasets with custom code `--model_args "pretrained=Qwen/Qwen2-0.5B-Instruct,trust_remote_code=True"`

A similar way to pass this to the lighteval CLI would be great.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.