huggingface / huggingface/optimum-nvidia
Load from local path?
- Dominant language
- Python
- Stars
- 1k
- Forks
- 103
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
I can load from HF repository, but I can't seem to load from a local path the way I can with hf.
Is this supported?
(I'm using -v with docker to mount a local dir, works fine in -it to, for example, provide access to .py files, so for example I load the below as 'python3 /home/work/models/test.py'
from argparse import ArgumentParser
from huggingface_hub import login
from optimum.nvidia import pipeline
if __name__ == "__main__":
token="...."
login(token)
model = pipeline("text-generation", model="/home/work/models/Meta-Llama-3-8B-Instruct-fp8", fp8=True)
out = model("What is the latest generation of Nvidia's GPUs?", max_new_tokens=128)
print(out)
Contributor guide
Assessment
This issue has not been assessed yet.