Typescript error while counting tokens
- Dominant language
- TypeScript
- Stars
- 59
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
When using the code shown in the readme :
```typescript
import Tokenizer, { models } from 'ai-tokenizer';
import { count } from 'ai-tokenizer/sdk';
import * as encoding from 'ai-tokenizer/encoding';
const model = models['openai/gpt-5'];
const tokenizer = new Tokenizer(encoding[model.encoding]);
const result = count({
tokenizer,
model,
messages: []
});
```
I have a Typescript error :
```text
Type 'import("node_modules/ai-tokenizer/dist/index").Tokenizer' is not assignable to type 'Tokenizer'.
Types have separate declarations of a private property 'name'.ts(2322)
```
My assumption would be that the `Tokenizer` class from `ai-tokenizer` and the `Tokenizer` class from `ai-tokenizer/sdk` have the same code but are separated class declarations, therefore are not type-compatible.
To fix this I guess there should be one and unique `Tokenizer` class used across all packages (but I can be mistaken).
In the meantime as a workaround I have to use the ugly :
```typescript
tokenizer as unknown as Tokenizer,
```
which is obviously dark magic.
Thanks for your time
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.