JapaneseTokenizer produces inconsistent tokens [LUCENE-9100]

We use `JapaneseTokenizer` on prod and seeing some inconsistent behavior. With this text:
`"マギアリス【単版話】 4話 (Unlimited Comics)"` I get different results if I insert space before `【` char. Here is the small code snippet demonstrating the case (note we use our own dictionary and connection costs):

```java
Analyzer analyzer = new Analyzer() {
`@Override`
protected TokenStreamComponents createComponents(String fieldName) {
// Tokenizer tokenizer = new JapaneseTokenizer(newAttributeFactory(), null, true, JapaneseTokenizer.Mode.SEARCH);
Tokenizer tokenizer = new JapaneseTokenizer(newAttributeFactory(), dictionaries.systemDictionary, dictionaries.unknownDictionary, dictionaries.connectionCosts, null, true, JapaneseTokenizer.Mode.SEARCH);
return new TokenStreamComponents(tokenizer, new LowerCaseFilter(tokenizer));
}
};
String text1 = "マギアリス【単版話】 4話 (Unlimited Comics)";
String text2 = "マギアリス 【単版話】 4話 (Unlimited Comics)"; //inserted space
try (TokenStream tokens = analyzer.tokenStream("field", new StringReader(text1))) {
CharTermAttribute chars = tokens.addAttribute(CharTermAttribute.class);
tokens.reset();
while (tokens.incrementToken()) {
System.out.println(chars.toString());
}
tokens.end();
} catch (IOException e) {
// should never happen with a StringReader
throw new RuntimeException(e);
}
```

Output is:

```java
//text1
マギ
アリス
単
版
話
4
話
unlimited
comics

//text2
マギア
リス
単
版
話
4
話
unlimited
comics
```

It looks like tokenizer doesn't view the punctuation (`【` is `Character.START_PUNCTUATION` type) as an indicator that there should be a token break, and somehow 【 punctuation char causes difference in the output. 

If I use the `JapaneseTokenizer` tokenizer then this problem doesn't manifest because it doesn't tokenize `マギアリス` into multiple tokens and outputs as is.

---
Migrated from [LUCENE-9100](https://issues.apache.org/jira/browse/LUCENE-9100) by Elbek Kamoliddinov, updated Jan 10 2020

Start with the JapaneseTokenizer reproducer in the issue, using SEARCH mode and the two supplied input strings, then compare token output with and without the space before 【. Trace how JapaneseTokenizer handles the 【 punctuation and dictionary-based segmentation. Done means the inconsistent tokenization is explained and covered by a regression test without breaking the existing tokenizer behavior.

Written by the indexing model from the issue text.