abetlen / abetlen/llama-cpp-python

Feature request: ability to tokenize a list of strings _or_ keep the tokenizer warm

オープン
#1,763 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

*Situation:* let's say you have a list of sentences you want to tokenize.

*Current workaround:*
```python
from llama_cpp import Llama

embedder = Llama.from_pretrained(repo_id="lm-kit/bge-m3-gguf", filename="*F16.gguf", embedding=True)
sentences = ["Hello world"] * 1000
sentences_tokens = [embedder.tokenize(sentence.encode()) for sentence in sentences]
# ↑ Each tokenize call has an overhead of 200-400ms
```

*Problem:* each call to `tokenize` appears to have an overhead of about 200-400ms. Which means tokenizing 1000 sentences will take 200-400 seconds 💥. Even tokenizing 10 sentences can take 4 seconds!

*Feature request*: either add the ability to tokenize a list of strings efficiently with `tokenize`, or add the ability to keep the tokenizer warm so that subsequent calls are not as slow as the first call.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。