coder / coder/ai-tokenizer

Typescript error while counting tokens

Open
#1 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
59
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Hello,

When using the code shown in the readme :

```typescript
import Tokenizer, { models } from 'ai-tokenizer';
import { count } from 'ai-tokenizer/sdk';
import * as encoding from 'ai-tokenizer/encoding';

const model = models['openai/gpt-5'];
const tokenizer = new Tokenizer(encoding[model.encoding]);
const result = count({
tokenizer,
model,
messages: []
});
```

I have a Typescript error :

```text
Type 'import("node_modules/ai-tokenizer/dist/index").Tokenizer' is not assignable to type 'Tokenizer'.
Types have separate declarations of a private property 'name'.ts(2322)
```

My assumption would be that the `Tokenizer` class from `ai-tokenizer` and the `Tokenizer` class from `ai-tokenizer/sdk` have the same code but are separated class declarations, therefore are not type-compatible.

To fix this I guess there should be one and unique `Tokenizer` class used across all packages (but I can be mistaken).

In the meantime as a workaround I have to use the ugly :

```typescript
tokenizer as unknown as Tokenizer,
```

which is obviously dark magic.

Thanks for your time

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.