dense-analysis / dense-analysis/neural
Feat: Add tokens counter
Open
enhancement
- Dominant language
- Vim Script
- Stars
- 510
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
## Overview
It would be useful to add the ability for neural to get the token count for some given input. This would help prevent initiating requests that accidentally go over the maximum token count for some given model source.
This will also be useful in situations where we want to extract the maximum possible response from a model via `request_token_num = model_max_token_len - context_tokens_len `
## Implementation
- The tokenizer should be appropriate for the respective model
- We should use an open-source (Ideally MIT) tokenizer that we can bundle to not require installing additional dependencies
Contributor guide
Assessment
This issue has not been assessed yet.