googleapis / googleapis/python-aiplatform

Counting tokens locally

Ouverte
#5,744 0 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
api: vertex-ai
Langage dominant
Python
Étoiles
905
Forks
465
Merge moyen
1 j 13 h
PR mergées (30 j)
44

Description

I was looking at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/get-token-count and there is a message showing that we can use local token counting (similarly to tiktoken).

"""
Important: Instead of using the countTokens API, we recommend that you use integrated tokenizer of the Vertex AI SDK for Python for getting token count. For details, see [List and count tokens](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token).
"""
When entering https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token it doesn't show any examples though. I have genai lib installed and couldn't find any related methods besides the API call method.

I heard that this lib / way of counting is being deprecated too:
```
from vertexai.preview import tokenization

model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")
```
So, is there a way to count tokens for gemini models locally? Any examples / docs about it?

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.