abetlen / abetlen/llama-cpp-python

Feature request: ability to tokenize a list of strings _or_ keep the tokenizer warm

Aberta
#1,763 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.4k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

*Situation:* let's say you have a list of sentences you want to tokenize.

*Current workaround:*
```python
from llama_cpp import Llama

embedder = Llama.from_pretrained(repo_id="lm-kit/bge-m3-gguf", filename="*F16.gguf", embedding=True)
sentences = ["Hello world"] * 1000
sentences_tokens = [embedder.tokenize(sentence.encode()) for sentence in sentences]
# ↑ Each tokenize call has an overhead of 200-400ms
```

*Problem:* each call to `tokenize` appears to have an overhead of about 200-400ms. Which means tokenizing 1000 sentences will take 200-400 seconds 💥. Even tokenizing 10 sentences can take 4 seconds!

*Feature request*: either add the ability to tokenize a list of strings efficiently with `tokenize`, or add the ability to keep the tokenizer warm so that subsequent calls are not as slow as the first call.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.