Refactoring of processor code

There are many processors which are doing very similar things. E.g. some of the 3rd party tokenizers. Even though the 3rd party interfaces might differ, there still can be code which would be doing similar things, e.g. checking for existing tokens in the input pack or creation of new `Token` entries to be added to the input pack. Can we combine these common parts to have less duplicate code? Maybe in the form of utility functions or common base class if necessary?

Also, find other ways to structure our processors as their number increases.

The issue does not name specific files, tests, or processor entry points. Start by inventorying the processor implementations and the third-party tokenizer integrations, then compare their input-pack token checks and Token creation paths. Done means the shared behavior is consolidated where appropriate and the growing processor structure is documented or made clearer without changing behavior.

Written by the indexing model from the issue text.