dotnet / dotnet/machinelearning

Expose Encoder in TiktokenTokenizer

Open
#7,313 9 comments 0 reactions 1 assignee Claimed by @tarekgh View on GitHub
area-Tokenizers enhancement in-pr needs-further-triage
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Hello, first of all thank your very much for this project!

**Is your feature request related to a problem? Please describe.**
Yes, it is.
Some of our clients may have outdated encodings on their client application.
We still want our clients to have access to new encodings even if their client application is not up to date, hence we want to serve the encoder dictionary from a server endpoint.

**A clear and concise description of what the problem is.**
The problem is that, currently, the Encoder property in TiktokenTokenizer is internal.

https://github.com/dotnet/machinelearning/blob/509032755b912e7bb0dd50c10a3172ead57965f3/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L998-L1001

**Describe the solution you'd like**
I would like to expose this Encoder property.
There seems to be the intent to expose this property at some point in the future.
https://github.com/dotnet/machinelearning/blob/509032755b912e7bb0dd50c10a3172ead57965f3/test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs#L732-L740

Maybe this is the time to do it, what do you think?

**Describe alternatives you've considered**
Maybe a separate method that does exactly what that test from above does using reflection.
Sounds like overkill and a lot of overhead though.
Exposing the property is probably the best way to deal with this.

**Additional context**
I'm sending a PR your way with the changes, feel free to ask for/make any modifications you think are necessary.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.