[smartcn] Some chinese text is not tokenized correctly with Chinese punctuation marks appended [LUCENE-7509]

Some chinese text is not tokenized correctly with Chinese punctuation marks appended.

e.g.
碧绿的眼珠 is tokenized as 碧绿|的|眼珠. Which is correct.

But
碧绿的眼珠，（with a Chinese punctuation appended )is tokenized as 碧绿|的|眼|珠，

The similar case happens when text with numbers appended.

e.g.
生活报8月4号 -->生活|报|8|月|4|号
生活报-->生活报

Test Sample:
public static void main(String[] args) throws IOException{
Analyzer analyzer = new SmartChineseAnalyzer(); /\* will load stopwords \*/
System.out.println("Sample1=======");
String sentence = "生活报8月4号";
printTokens(analyzer, sentence);
sentence = "生活报";
printTokens(analyzer, sentence);
System.out.println("Sample2=======");

sentence = "碧绿的眼珠，";
printTokens(analyzer, sentence);
sentence = "碧绿的眼珠";
printTokens(analyzer, sentence);

analyzer.close();

}

private static void printTokens(Analyzer analyzer, String sentence) throws IOException{
System.out.println("sentence:" + sentence);
TokenStream tokens = analyzer.tokenStream("dummyfield", sentence);
tokens.reset();
CharTermAttribute termAttr = (CharTermAttribute) tokens.getAttribute(CharTermAttribute.class);
while (tokens.incrementToken()) {
System.out.println(termAttr.toString());
}
tokens.close();
}

Output:
Sample1=======
sentence:生活报8月4号
生活
报
8
月
4
号
sentence:生活报
生活报
Sample2=======
sentence:碧绿的眼珠，
碧绿
的
眼
珠
sentence:碧绿的眼珠
碧绿
的
眼珠

---
Migrated from [LUCENE-7509](https://issues.apache.org/jira/browse/LUCENE-7509) by peina, updated Dec 05 2016
Environment:
```
Mac OS X 10.10
```

Start with SmartChineseAnalyzer and reproduce the examples in the issue, comparing tokenization with and without the appended Chinese punctuation or numbers. Trace the analyzer's tokenization behavior to identify why the appended characters change the segmentation. Done means the affected examples produce consistent, correct tokens in both forms, with regression coverage added where the relevant tests are located.

Written by the indexing model from the issue text.