abetlen / abetlen/llama-cpp-python

Add n_keep parameter to LLama constructor to enable Streaming-LLM

Open
#954 1 comment 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

A recent [paper](https://arxiv.org/pdf/2309.17453.pdf) by Meta/MIT/CMU proposed [StreamingLLM](https://github.com/mit-han-lab/streaming-llm/), a simple yet efficient solution to enable "infinite" context. Better yet, the implementation in llama.cpp is as trivial as changing the `n_keep` value with option `--keep` as discussed [in this issue](https://github.com/ggerganov/llama.cpp/issues/3440). Unfortunately, the high-level API of llama-cpp-python does not support the `keep`/`n_keep` parameter.

It should be simple to add the parameter to the high-level API, ideally in the constructor for class `Llama` and to pass it along to function `llama_cpp.llama_load_model_from_file` as part of parameter `lparams` [here](https://github.com/abetlen/llama-cpp-python/blob/f3b844ed0a139fc5799d6e515e9d1d063c311f97/examples/low_level_api/low_level_api_llama_cpp.py#L16C19-L16C45).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.