huggingface / huggingface/optimum-nvidia

Load from local path?

Open
#126 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
103
Avg merge
2m
Merged PRs (30d)
1

Description

I can load from HF repository, but I can't seem to load from a local path the way I can with hf.
Is this supported?
(I'm using -v with docker to mount a local dir, works fine in -it to, for example, provide access to .py files, so for example I load the below as 'python3 /home/work/models/test.py'

from argparse import ArgumentParser
from huggingface_hub import login
from optimum.nvidia import pipeline

if __name__ == "__main__":

token="...."
login(token)

model = pipeline("text-generation", model="/home/work/models/Meta-Llama-3-8B-Instruct-fp8", fp8=True)
out = model("What is the latest generation of Nvidia's GPUs?", max_new_tokens=128)
print(out)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.