googleapis / googleapis/python-aiplatform
Counting tokens locally
- 主要语言
- Python
- 星标
- 905
- 派生
- 465
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 44
描述
I was looking at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/get-token-count and there is a message showing that we can use local token counting (similarly to tiktoken).
"""
Important: Instead of using the countTokens API, we recommend that you use integrated tokenizer of the Vertex AI SDK for Python for getting token count. For details, see [List and count tokens](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token).
"""
When entering https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token it doesn't show any examples though. I have genai lib installed and couldn't find any related methods besides the API call method.
I heard that this lib / way of counting is being deprecated too:
```
from vertexai.preview import tokenization
model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)
contents = "Hello World!"
result = tokenizer.count_tokens(contents)
print(f"{result.total_tokens = :,}")
```
So, is there a way to count tokens for gemini models locally? Any examples / docs about it?
贡献指南
调研方向
Start with the linked Vertex AI token-counting and list-token documentation, then inspect the installed Python genai library for the current Gemini tokenizer entry point. Confirm whether local counting is supported and whether the older vertexai.preview.tokenization example is deprecated. Document the supported method with a working example, or clarify the limitation if no local API exists.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- documentation, machine-learning
- Issue 类型
- 文档
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 30/100