googleapis / googleapis/python-aiplatform

Counting tokens locally

Abierto
#5,744 0 comentarios 1 reacción 0 asignados Ver en GitHub
api: vertex-ai
Lenguaje dominante
Python
Estrellas
905
Forks
465
Merge medio
1 d 13 h
PR fusionados (30 d)
44

Descripción

I was looking at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/get-token-count and there is a message showing that we can use local token counting (similarly to tiktoken).

"""
Important: Instead of using the countTokens API, we recommend that you use integrated tokenizer of the Vertex AI SDK for Python for getting token count. For details, see [List and count tokens](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token).
"""
When entering https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token it doesn't show any examples though. I have genai lib installed and couldn't find any related methods besides the API call method.

I heard that this lib / way of counting is being deprecated too:
```
from vertexai.preview import tokenization

model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")
```
So, is there a way to count tokens for gemini models locally? Any examples / docs about it?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.