huggingface / huggingface/evaluate

Cannot access gated repositories in Perplexity measurements module

Open
#680 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Since accessing gated models requires authentication by passing either a hf_token or using the login() method.

The **measurements/perplexity/README.md** file doesn't mention how to authenticate for gated models, and I've tried passing the token via model configuration parameters directly to the **compute()** function. However, it seems there's no mechanism in place to capture **kwargs and pass them down to from_pretrained() within the current implementation:

https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/measurements/perplexity/perplexity.py#L104-L106

https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/src/evaluate/module.py#L467

An easy way would be just use login function before calling compute function

```python
from huggingface_hub import login

hf_token = os.environ["HF_TOKEN"]
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

perplexity = load("perplexity", module_type="measurement")

login(token = hf_token)

results = perplexity.compute(
data=[first_question, second_question],
model_id=model_id,
)

```

But this doesn't offer any flexibility for passing additional model parameters to from_pretrained().
It would be helpful if calling compute() could accept and propagate such kwargs to model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.