microsoft / microsoft/LLMLingua
[Question]: Running LLMLingua with GGUF models
Open
@iofu728 is already working on this.
Since Mar 7, 2024.
question
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 428
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 1
Description
Describe the issue
Hi, trying to make it run properly with GGUF models (i.e. CPU only) due to RAM restriction,
Trying to use it as
compressor = PromptCompressor(
device_map="cpu",
model_name="TheBloke/Llama-2-7B-GGUF",
model_config={ 'model_file': "llama-2-7b.Q4_K_M.gguf", 'model_type': "llama", 'gpu_layers': 0 }
)
but need to somehow push some code for using llama-cpp so I can load it properly (otherwise stops at tokenizer)
anyone already done this ? is it planned to be supported ? or would any have an advice on how to proceed
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.