tokenizer supports preserving delimiters [LUCENE-9308]

currently there s no way to preserve the delimiter in tokenizer, because the basic tokenizer like CharTokenizer ignore them.

this s to make the basic tokenizer more customizable 

e.g. "mac_book_pro" -> [mac_, book_, pro]

---
Migrated from [LUCENE-9308](https://issues.apache.org/jira/browse/LUCENE-9308) by yin Lin

Start from Lucene’s basic tokenizer and trace how it currently discards delimiters, using the mac_book_pro example as the expected behavior. Done means the tokenizer can be customized to preserve delimiters and produces the illustrated token sequence, with coverage for that behavior.

Written by the indexing model from the issue text.