abetlen / abetlen/llama-cpp-python

Add n_keep parameter to LLama constructor to enable Streaming-LLM

未关闭
#954 1 条评论 4 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

A recent [paper](https://arxiv.org/pdf/2309.17453.pdf) by Meta/MIT/CMU proposed [StreamingLLM](https://github.com/mit-han-lab/streaming-llm/), a simple yet efficient solution to enable "infinite" context. Better yet, the implementation in llama.cpp is as trivial as changing the `n_keep` value with option `--keep` as discussed [in this issue](https://github.com/ggerganov/llama.cpp/issues/3440). Unfortunately, the high-level API of llama-cpp-python does not support the `keep`/`n_keep` parameter.

It should be simple to add the parameter to the high-level API, ideally in the constructor for class `Llama` and to pass it along to function `llama_cpp.llama_load_model_from_file` as part of parameter `lparams` [here](https://github.com/abetlen/llama-cpp-python/blob/f3b844ed0a139fc5799d6e515e9d1d063c311f97/examples/low_level_api/low_level_api_llama_cpp.py#L16C19-L16C45).

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。