abetlen / abetlen/llama-cpp-python

LlamaCPP Usage

未关闭
#1,035 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
question
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

i use this guide for develop a RAG system: https://github.com/marklysze/LlamaIndex-RAG-WSL-CUDA/blob/master/LlamaIndex_Mixtral_8x7B-RAG.ipynb

i use the following code for use my local llm:

```
llm = LlamaCPP(
model_url=None, # We'll load locally.
# model_path='./Models/mistral-7b-instruct-v0.1.Q6_K.gguf', # 6-bit model
model_path="my_local_llm_path",
temperature=0.1,
max_new_tokens=1024, # Increasing to support longer responses
context_window=8192, # Mistral7B has an 8K context-window
generate_kwargs={},
# set to at least 1 to use GPU
model_kwargs={"n_gpu_layers": 40}, # 40 was a good amount of layers for the RTX 3090, you may need to decrease yours if you have less VRAM than 24GB
# messages_to_prompt=messages_to_prompt,
# completion_to_prompt=completion_to_prompt,
messages_to_prompt=system_prompt,
completion_to_prompt=query_wrapper_prompt,
#system_prompt=system_prompt,
#query_wrapper_prompt=query_wrapper_prompt,
verbose=True
)
```
Can anyone give me information about LlamaCPP structure? What should i do for use my local model?

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。