huggingface / huggingface/evaluate
Cannot access gated repositories in Perplexity measurements module
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Since accessing gated models requires authentication by passing either a hf_token or using the login() method.
The **measurements/perplexity/README.md** file doesn't mention how to authenticate for gated models, and I've tried passing the token via model configuration parameters directly to the **compute()** function. However, it seems there's no mechanism in place to capture **kwargs and pass them down to from_pretrained() within the current implementation:
https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/measurements/perplexity/perplexity.py#L104-L106
https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/src/evaluate/module.py#L467
An easy way would be just use login function before calling compute function
```python
from huggingface_hub import login
hf_token = os.environ["HF_TOKEN"]
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
perplexity = load("perplexity", module_type="measurement")
login(token = hf_token)
results = perplexity.compute(
data=[first_question, second_question],
model_id=model_id,
)
```
But this doesn't offer any flexibility for passing additional model parameters to from_pretrained().
It would be helpful if calling compute() could accept and propagate such kwargs to model.
Contributor guide
Assessment
This issue has not been assessed yet.