dense-analysis / dense-analysis/neural

Feat: Add tokens counter

Open
#41 0 comments 0 reactions 1 assignee Claimed by @Angelchev View on GitHub
enhancement
Dominant language
Vim Script
Stars
510
Forks
20
PR merge metrics
No merged PRs in 30d

Description

## Overview

It would be useful to add the ability for neural to get the token count for some given input. This would help prevent initiating requests that accidentally go over the maximum token count for some given model source.

This will also be useful in situations where we want to extract the maximum possible response from a model via `request_token_num = model_max_token_len - context_tokens_len `

## Implementation
- The tokenizer should be appropriate for the respective model
- We should use an open-source (Ideally MIT) tokenizer that we can bundle to not require installing additional dependencies

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.