abetlen / abetlen/llama-cpp-python

Feature request: ability to tokenize a list of strings _or_ keep the tokenizer warm

未关闭
#1,763 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

*Situation:* let's say you have a list of sentences you want to tokenize.

*Current workaround:*
```python
from llama_cpp import Llama

embedder = Llama.from_pretrained(repo_id="lm-kit/bge-m3-gguf", filename="*F16.gguf", embedding=True)
sentences = ["Hello world"] * 1000
sentences_tokens = [embedder.tokenize(sentence.encode()) for sentence in sentences]
# ↑ Each tokenize call has an overhead of 200-400ms
```

*Problem:* each call to `tokenize` appears to have an overhead of about 200-400ms. Which means tokenizing 1000 sentences will take 200-400 seconds 💥. Even tokenizing 10 sentences can take 4 seconds!

*Feature request*: either add the ability to tokenize a list of strings efficiently with `tokenize`, or add the ability to keep the tokenizer warm so that subsequent calls are not as slow as the first call.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。