dotnet / dotnet/machinelearning

Add max number of token to count to the TokenCount function

Open
#7,391 1 comment 0 reactions 0 assignees View on GitHub
area-Tokenizers enhancement in-pr
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Is your feature request related to a problem? Please describe.

When processing large inputs, I need to count the tokens to ensure they do not exceed the maximum allowed for my request before sending it. Currently, the existing interfaces require counting all tokens in the input, even if I only need to know if the count exceeds a specific number. This results in unnecessary processing. It would be more efficient to have the ability to stop counting once a certain threshold is reached.

Describe the solution you'd like

Introduce an optional parameter in the CountTokens method that allows setting a maximum number of tokens to count. This way, the counting process can terminate early if the specified limit is reached, saving processing time and resources. This parameter should be added to the existing parameters for the counting method

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.