huggingface / huggingface/optimum-nvidia

Error for gated model access despite valid HF_TOKEN

Open
#142 0 comments 1 reaction 1 assignee Claimed by @mfuntowicz View on GitHub
Dominant language
Python
Stars
1k
Forks
103
Avg merge
2m
Merged PRs (30d)
1

Description

I am trying to run a gated model through the pipeline API but I get a gated model access error despite having the HF_TOKEN env var set.

```
>>> from optimum.nvidia.pipelines import pipeline as optimum_pipeline
>>> fast_pipe = optimum_pipeline('text-generation', 'meta-llama/Llama-2-70b-chat-hf', tp=2, use_fp8=True)
[...]
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.10/dist-packages/optimum/nvidia/pipelines/__init__.py", line 93, in pipeline
raise RuntimeError(
RuntimeError: Failed to instantiate the pipeline inferring the task for model meta-llama/Llama-2-70b-chat-hf: 401 Client Error. (Request ID: Root=1-66859d97-068675f635fb5d9f4e0b36b6;10764411-e918-4f91-8a97-ca114e65ea79)

Cannot access gated repo for url https://huggingface.co/api/models/meta-llama/Llama-2-70b-chat-hf.
Access to model meta-llama/Llama-2-70b-chat-hf is restricted. You must be authenticated to access it.
>>> import os
>>> os.environ["HF_TOKEN"]
'***' # this is my access token that has access to the model
```

Running with the transformers pipeline succeeds in downloading the checkpoint

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.