The tokenstream create by SmartChineseAnalyzer can't reset [LUCENE-3834]

That is because the field input in class SentenceTokenizer isn't reset after we call the method reset().

They are two input field,one is from Tokenizer and another is from TokenFilter,if we need to reset a tokenstream created by SmartChineseAnalyzer, both them need reset.This bug is because of the author forget reset input field in class SentenceTokenizer .

class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer

oringal code

public final class SentenceTokenizer extends Tokenizer {
....
`@Override`
public void reset() throws IOException {
super.reset();
tokenStart = tokenEnd = 0;
}

...
}

this method should changes as follow

public void reset() throws IOException {
super.reset();
/**should reset input**/
if (input.markSupported())
input.reset();
tokenStart = tokenEnd = 0;
}

---
Migrated from [LUCENE-3834](https://issues.apache.org/jira/browse/LUCENE-3834) by dingjin, updated Mar 01 2012

Read org.apache.lucene.analysis.cn.smart.SentenceTokenizer, starting with its reset() method and the Tokenizer and TokenFilter input fields described in the issue. Verify that resetting a SmartChineseAnalyzer token stream handles both inputs and resets tokenStart and tokenEnd; done means repeated reset and reuse no longer retain prior input state.

Written by the indexing model from the issue text.