abetlen / abetlen/llama-cpp-python
Support for LLMLingua
オープン
enhancement
- 主要言語
- Python
- スター
- 10.6k
- フォーク
- 1.4k
- PR マージ指標
- PR 指標を取得中
説明
microsoft/LLMLingua seems to be an interesting project. It's essentially (lossy) prompt compression, and you can use any HF model with it currently, including GPTQ. I think it would be useful to have llama.cpp supported via llama-cpp-python, since prompt compression would be useful for both CPU and GPU users, and especially alongside llama.cpp itself.
I was trying to implement llama-cpp-python for inference, but got stuck on needing an attention mask (perhaps I missed something). Any ideas on how to go about this?
microsoft/LLMLingua#41
コントリビューションガイド
評価
この issue はまだ評価されていません。