Can I retokenize at the start of a training pipeline?
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
I need to perform a lot of _retokenization_ before running a _training pipeline_, but from the doc I cannot understand if that is possible and, if yes, how to specify that in the config file.
In https://github.com/explosion/spaCy/discussions/5921 , @svlandeg showed how to deal with a similar issue, in the case at hand (i.e. training NER), without adding a custom component; thus, she didn't explicitly answer the original question and my question.
As I explained in issue https://github.com/explosion/spaCy/issues/13248 and, more extensively, in discussion https://github.com/explosion/spaCy/discussions/7146 , I'm struggling to develop a viable tokenizer for the _Arabic language_. For doing that, I think I need both to extend the data (the configuration files) in the current implementation of the tokenizer and to add a considerable amount of post-processing.
In the past, I've implemented the post-processing with some _Cython_ code and I began to get significantly improved results from the _data debug_ and _train_ commands. Then, I installed _spaCy from source_, but in this case I wasn't able to integrate my Cython code with the spaCy codebase, more precisely to import _tokenizer.Tokenizer_ and _vocab.Vocab_.
Now, I guess that being able to put a component just after the spaCy _Tokenizer_ in the training pipeline (and in the production pipeline) would be much cleaner and probably more efficient.
Could somebody answer my question and/or suggest a solution for my problem? Thanks in advance!
Contributor guide
Assessment
This issue has not been assessed yet.