Feature Requests on Tokenizers - Serialization / Model-Fitting / Custom-Vocabulary

Hello,

I have been starting using TensorFlow Text in reaction to the deprecation of the TextEncoder classes in TensorFlow Datasets, with the hope that I could easily convert my code to switch from the latter to the former. Long story short: it proves hard.

There are some really exciting features in TensorFlow Text, and I would like to thank everybody who has been working on that, but as a user, after a few hours of testing things and reading the python part of the source code, I feel like a few things are lacking to make their adoption simple. So, I would like to humbly provide some feedback and request some features as to the Tokenizer classes, for the implementation of which I would actually be willing to give a hand.

(0) Tokenizer Serialization

It would be really nice to be able to dump and reload a Tokenizer without having to ponder about all the (very cool, modularity-yielding) instantiation arguments. I believe that the easiest would probably just be to mimic the get_config / from_config keras API, basically returning dict of serializable keyword arguments.

To be fair, my main issue at the moment is with the SentencepieceTokenizer (as well as the Wordpiece and Bert ones), whose main instantiation argument is made inaccessible after intantiation (at least using the publicly exposed attributes and methods). Being able to access the proto of a built tokenizer and export it back (either as bytes or using a hex-encoding trick to be writable to json) would be really nice.

(1) Tokenizer Fitting

At the moment, if I want to use a SentencepieceTokenizer on my data, I have to first use the sentencepiece module, use it to fit a model on my data (which is semi-easy: the tool is amazing, but the documentation is so split apart that it can take time to manage using it), read one of the output files's bytes (the 'proto', which is not clearly documented to be so) and instantiate a tokenizer from it (which to be built again will require remembering what the 'proto', but that is my former point).

I think that it would be neat to have a SentencepieceTokenizer.build_from_data method (on the model of SubwordTextEncoder.build_from_corpus) that would do the job of interfacing the sentencepiece package, gathering the proto and (last but not least) documenting the former package's options (at least the main ones). This could then serve for future implementations supporting alternative tokenizer-fitting algorithms, including simple no-subwords frequency-based ones (which are my last point).

(2) Custom-Vocabulary Tokenizers

Finally, it would be nice to be able to instantiate a tokenizer that works using an arbitrary input vocabulary (like the former TokenTextEncoder one). I know that this would be partly redundant with tf.keras.layers.preprocessing.experimental.TextVectorization, but it probably would not be hard to achieve, and would offer a non-layer alternative to the former. I am actually wondering to what extent a SentencepieceTokenizer subclass without sampling options could simply be used (with a wrapper to convert input vocabulary list to the proper proto format).

I am aware that building TensorFlow Text takes time, and I apologize if this is is any way redundant with existing roadmaps that I would not be aware of. I am truly looking forward to making TF-Text part of my tools, and am simply hoping that feedback and possible contribution through PRs (once the former points have been discussed, adjusted and validated) can help avoid having users (myself including) forced to write custom framework hacks that end up not being easily sharable nor maintainable.

Thank you for any time and effort put in reviewing what precedes. I hope it can result in actions, and am looking forward to contribute if needed.

Start by reviewing the Python tokenizer APIs mentioned in the issue, especially SentencepieceTokenizer, Wordpiece, Bert, and SubwordTextEncoder. Compare the requested serialization, model-fitting, and custom-vocabulary capabilities and determine which request can be scoped independently. Done requires an agreed feature scope and corresponding implementation plan.