JetBrains-Research / JetBrains-Research/TestSpark
OpenAI: Count tokens without sending a request to API
Open
new feature
- Dominant language
- Kotlin
- Stars
- 88
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
There is a Python tool called `tiktoken`, which is an official tokenizer that counts the number of tokens in messages to be sent. It allows you to identify token overflows without sending a request.
If the same official library exists but in Kotlin, we can utilize it; it will speed up the test generation workflow, as there will be no need for HTTP request calls if we already know that the request will fail.
**See:** https://platform.openai.com/docs/guides/text-generation/managing-tokens
Contributor guide
Assessment
This issue has not been assessed yet.