abetlen / abetlen/llama-cpp-python

Question: How to keep longer conversations without breaking the code?

未关闭
#1,139 3 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

I want to have longer conversations with the model, but as I understand, the number of tokens in the prompt message + the number of tokens generated by the model should be less than the context size (`n_ctx`). As the conversation develops, more tokens are stored in `messages`, which would eventually result in the following error:

`ValueError: Requested tokens (2083) exceed context window of 2048`

I am new to all this, but I think the way it is handled in `llama.cpp` is that it is managed with the `n_keep` parameter ([here](https://github.com/ggerganov/llama.cpp/discussions/1838) is a helpful discussion). However, I can't find how to avoid the mentioned error using `llama-cpp-python`. I think the chat context should be reset at some point, because even if we use `n_keep` (which I don't know how to do using this repo), the context window would still get filled up.

So, my question is how can I handle longer chats when using `create_chat_completion` and avoid getting the mentioned error using some functionality similar to `n_keep` combined with resetting `total_tokens` and emptying the current context when filled up?

Thank you!

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。