dotnet / dotnet/machinelearning
Tokenizers: read the tokenizer.json 'decoder' section in CreateFromTokenizerJson
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### Background
`SentencePieceTokenizer.CreateFromTokenizerJson` (added in #7625) loads a Hugging Face `tokenizer.json` for Unigram models. It reads `model`, `normalizer`, `pre_tokenizer`, `added_tokens`, and `post_processor`, but **does not read the `decoder` section**.
This was called out in review of #7625: it is fine for the standard HF Unigram/Metaspace configurations the current tests cover (decoding is driven by the SentencePiece model semantics), but it means non-standard `decoder` chains in a `tokenizer.json` are ignored rather than honored.
### Ask
Support reading the `tokenizer.json` `decoder` section when a real model requires a decoder chain that differs from the implied SentencePiece/Metaspace decoding, e.g. `Sequence`, `Replace`, `Strip`, `ByteFallback`, `Metaspace`, `Fuse` decoders. Until then, the loader silently relies on the model''s built-in decode behavior.
### Notes
- Related items from the same review were addressed in #7625: accepting a null `unk_id`, and rejecting unrecognized `pre_tokenizer` types with `NotSupportedException`.
- No known real Unigram model in the survey required a custom decoder chain, so this is a completeness gap rather than a blocking bug.
_Filed as a follow-up to review feedback on #7625._
Contributor guide
Assessment
This issue has not been assessed yet.