abetlen / abetlen/llama-cpp-python

Include usage key in create_completion when streaming

Abierto
#1,498 2 comentarios 4 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

**Is your feature request related to a problem? Please describe.**
Since `create_completion` may yield text chunks comprised of multiple tokens per yield (e.g. in the case of multi-byte Unicode characters), counting the number of yields may not equal the number of tokens actually generated by a model. To accurately get the usage statistics of a streamed completion, one has to run the final text through the tokenizer again, despite `create_completion` already tracking the number of tokens generated by the model.

**Describe the solution you'd like**
When `stream=True` in `create_completion`, the final chunk yielded should include the usage statistics in the `'usage'` key.

**Describe alternatives you've considered**
- Saving full generated text and running it through the tokenizer again (seems wasteful)
- Counting the number of yields and hoping we don't have any multi-byte characters (hacky and fragile)

**Additional context**
The OpenAI API has recently added similar support in their streaming API with the `stream_options` key: https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.