dotnet / dotnet/machinelearning

Enable custom token counting

Open
#7,263 0 comments 0 reactions 0 assignees View on GitHub
area-Tokenizers enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Currently, Tokenizer returns counts strictly based on tokens.

However, there are scenarios where library authors may want / need to implement their own custom token counting function.

One such scenario is providing token counts for image inputs. In such cases, AI service providers provide an arbitrary way of calculating cost based on a fixed token count.

| Provider | Cost calculations | Does tokenization | Link |
| --- | --- | --- | --- |
| OpenAI | Fixed | No | https://platform.openai.com/docs/guides/vision/calculating-costs |
| Claude | Fixed | No | https://docs.anthropic.com/en/docs/build-with-claude/vision#calculate-image-costs |
| Gemini | Fixed | No | https://ai.google.dev/gemini-api/docs/tokens?lang=python#multimodal-tokens |
| Cohere | N/A | N/A | https://docs.cohere.com/docs/tokens |
| Mistral | N/A | N/A | https://docs.mistral.ai/guides/tokenization/#tokens-count |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.