dotnet / dotnet/machinelearning

Tokenizers: read the tokenizer.json 'decoder' section in CreateFromTokenizerJson

Open
#7,660 1 comment 0 reactions 0 assignees View on GitHub
area-Tokenizers enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### Background

`SentencePieceTokenizer.CreateFromTokenizerJson` (added in #7625) loads a Hugging Face `tokenizer.json` for Unigram models. It reads `model`, `normalizer`, `pre_tokenizer`, `added_tokens`, and `post_processor`, but **does not read the `decoder` section**.

This was called out in review of #7625: it is fine for the standard HF Unigram/Metaspace configurations the current tests cover (decoding is driven by the SentencePiece model semantics), but it means non-standard `decoder` chains in a `tokenizer.json` are ignored rather than honored.

### Ask

Support reading the `tokenizer.json` `decoder` section when a real model requires a decoder chain that differs from the implied SentencePiece/Metaspace decoding, e.g. `Sequence`, `Replace`, `Strip`, `ByteFallback`, `Metaspace`, `Fuse` decoders. Until then, the loader silently relies on the model''s built-in decode behavior.

### Notes

- Related items from the same review were addressed in #7625: accepting a null `unk_id`, and rejecting unrecognized `pre_tokenizer` types with `NotSupportedException`.
- No known real Unigram model in the survey required a custom decoder chain, so this is a completeness gap rather than a blocking bug.

_Filed as a follow-up to review feedback on #7625._

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.