abetlen / abetlen/llama-cpp-python

Help needed with loading 'TheBloke/Mistral-7B-Instruct-v0.1-GGUF' model using `llama-cpp-python`

Open
#915 8 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

Hello,

I'm currently working on a project that requires the use of the `TheBloke/Mistral-7B-Instruct-v0.1-GGUF` model, which is in the GGUF format. I've tried using the Hugging Face library to load this model, but it seems that the library does not support the GGUF format.

I've also tried using the `ctransformers` library, but I've encountered some issues with it as well. Therefore, I'm considering using the llama-cpp-python library instead.

However, I'm having trouble understanding how to use the llama-cpp-python library to load the 'TheBloke/Mistral-7B-Instruct-v0.1-GGUF' model. The documentation for the llama-cpp-python library is not very detailed, and there are no specific examples of how to use this library to load a model from the Hugging Face Model Hub.

I would greatly appreciate if you could provide some guidance on how to use the llama-cpp-python library to load the `TheBloke/Mistral-7B-Instruct-v0.1-GGUF` model. Specifically, I would like to know how to install the library, how to import it in my Python code, and how to use it to load the model.

Thank you in advance for your help.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.