Unifying similar test functions together

There are many processors which are testing very similar things. E.g. all the tokenizer tests are testing the tokens created by those processors which are same for our test cases. Can we combine these common parts to have less duplicate code?

Also, find other ways to structure our test framework as the number of processors increase.

Start by inventorying the processor tests, especially the tokenizer tests, and compare their repeated token assertions. Review how the test framework is structured as processors increase. Done means common test logic is consolidated without changing coverage and the test suite still passes.

Written by the indexing model from the issue text.