JetBrains-Research / JetBrains-Research/TestSpark

OpenAI: Count tokens without sending a request to API

Open
#283 0 comments 0 reactions 0 assignees View on GitHub
new feature
Dominant language
Kotlin
Stars
88
Forks
24
PR merge metrics
No merged PRs in 30d

Description

There is a Python tool called `tiktoken`, which is an official tokenizer that counts the number of tokens in messages to be sent. It allows you to identify token overflows without sending a request.

If the same official library exists but in Kotlin, we can utilize it; it will speed up the test generation workflow, as there will be no need for HTTP request calls if we already know that the request will fail.

**See:** https://platform.openai.com/docs/guides/text-generation/managing-tokens

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.