googleapis / googleapis/python-aiplatform

Counting tokens locally

未关闭
#5,744 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
api: vertex-ai
主要语言
Python
星标
905
派生
465
平均合并
1 天 13 小时
30 天内合并 PR
44

描述

I was looking at https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/get-token-count and there is a message showing that we can use local token counting (similarly to tiktoken).

"""
Important: Instead of using the countTokens API, we recommend that you use integrated tokenizer of the Vertex AI SDK for Python for getting token count. For details, see [List and count tokens](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token).
"""
When entering https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/list-token it doesn't show any examples though. I have genai lib installed and couldn't find any related methods besides the API call method.

I heard that this lib / way of counting is being deprecated too:
```
from vertexai.preview import tokenization

model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")
```
So, is there a way to count tokens for gemini models locally? Any examples / docs about it?

贡献指南

打开贡献指南

调研方向

Start with the linked Vertex AI token-counting and list-token documentation, then inspect the installed Python genai library for the current Gemini tokenizer entry point. Confirm whether local counting is supported and whether the older vertexai.preview.tokenization example is deprecated. Document the supported method with a working example, or clarify the limitation if no local API exists.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
documentation, machine-learning
Issue 类型
文档
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
30/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。