explosion / explosion/spaCy

Can I retokenize at the start of a training pipeline?

Open
#13,484 0 comments 0 reactions 0 assignees View on GitHub
feat / tokenizer
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

I need to perform a lot of _retokenization_ before running a _training pipeline_, but from the doc I cannot understand if that is possible and, if yes, how to specify that in the config file.
In https://github.com/explosion/spaCy/discussions/5921 , @svlandeg showed how to deal with a similar issue, in the case at hand (i.e. training NER), without adding a custom component; thus, she didn't explicitly answer the original question and my question.

As I explained in issue https://github.com/explosion/spaCy/issues/13248 and, more extensively, in discussion https://github.com/explosion/spaCy/discussions/7146 , I'm struggling to develop a viable tokenizer for the _Arabic language_. For doing that, I think I need both to extend the data (the configuration files) in the current implementation of the tokenizer and to add a considerable amount of post-processing.
In the past, I've implemented the post-processing with some _Cython_ code and I began to get significantly improved results from the _data debug_ and _train_ commands. Then, I installed _spaCy from source_, but in this case I wasn't able to integrate my Cython code with the spaCy codebase, more precisely to import _tokenizer.Tokenizer_ and _vocab.Vocab_.

Now, I guess that being able to put a component just after the spaCy _Tokenizer_ in the training pipeline (and in the production pipeline) would be much cleaner and probably more efficient.
Could somebody answer my question and/or suggest a solution for my problem? Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.