WordDelimiterFilter bug [LUCENE-6991]

I was preparing analyzer which contains WordDelimiterFilter and I realized it sometimes gives results different then expected.

I prepared a short test which shows the problem. I haven't used Lucene tests for this but this doesn't matter for showing the bug.

```Java
String urlIndexed = "144.214.37.14 - - [05/Jun/2013:08:39:27 +0000] \"GET /products/key-phrase-extractor/ HTTP/1.1\"" +
" 200 3437 http://www.google.com/url?sa=t&rct=j&q=&esrc=s&" +
"source=web&cd=15&cad=rja&ved=0CEgQFjAEOAo&url=http%3A%2F%2Fwww.sematext.com%2Fproducts%2Fkey-" +
"phrase-extractor%2F&ei=TPOuUbaWM-OKiQfGxIGYDw&usg=AFQjCNGwYAFYg_M3EZnp2eEWJzdvRrVPrg&sig2" +
"=oYitONI2EIZ0CQar7Ej8HA&bvm=bv.47380653,d.aGc\" \"Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:20.0) " +
"Gecko/20100101 Firefox/20.0\"";

List tokens1 = new ArrayList();
List tokens2 = new ArrayList();
WhitespaceAnalyzer analyzer = new WhitespaceAnalyzer();
TokenStream tokenStream = analyzer.tokenStream("test", urlIndexed);
tokenStream = new WordDelimiterFilter(tokenStream,
WordDelimiterFilter.GENERATE_WORD_PARTS |
WordDelimiterFilter.CATENATE_WORDS |
WordDelimiterFilter.SPLIT_ON_CASE_CHANGE,
null);
CharTermAttribute charAttrib = tokenStream.addAttribute(CharTermAttribute.class);
tokenStream.reset();
while(tokenStream.incrementToken()) {
tokens1.add(charAttrib.toString());
System.out.println(charAttrib.toString());
}
tokenStream.end();
tokenStream.close();

urlIndexed = "144.214.37.14 - - [05/Jun/2013:08:39:27 +0000] \"GET /products/key-phrase-extractor/ HTTP/1.1\"" +
" 200 3437 \"http://www.google.com/url?sa=t&rct=j&q=&esrc=s&" +
"source=web&cd=15&cad=rja&ved=0CEgQFjAEOAo&url=http%3A%2F%2Fwww.sematext.com%2Fproducts%2Fkey-" +
"phrase-extractor%2F&ei=TPOuUbaWM-OKiQfGxIGYDw&usg=AFQjCNGwYAFYg_M3EZnp2eEWJzdvRrVPrg&sig2" +
"=oYitONI2EIZ0CQar7Ej8HA&bvm=bv.47380653,d.aGc\" \"Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:20.0) " +
"Gecko/20100101 Firefox/20.0\"";

System.out.println("\n\n====\n\n");
tokenStream = analyzer.tokenStream("test", urlIndexed);
tokenStream = new WordDelimiterFilter(tokenStream,
WordDelimiterFilter.GENERATE_WORD_PARTS |
WordDelimiterFilter.CATENATE_WORDS |
WordDelimiterFilter.SPLIT_ON_CASE_CHANGE,
null);
charAttrib = tokenStream.addAttribute(CharTermAttribute.class);
tokenStream.reset();
while(tokenStream.incrementToken()) {
tokens2.add(charAttrib.toString());
System.out.println(charAttrib.toString());
}
tokenStream.end();
tokenStream.close();

assertEquals(Joiner.on(",").join(tokens1), Joiner.on(",").join(tokens2));
```

---
Migrated from [LUCENE-6991](https://issues.apache.org/jira/browse/LUCENE-6991) by Pawel Rog

Start by reproducing the Java snippet using WordDelimiterFilter and compare the two token streams produced from the nearly identical URL strings. Read the WordDelimiterFilter implementation and its existing tests, if present. Done means the input difference is explained and the filter has a regression test covering the corrected, deterministic behavior.

Written by the indexing model from the issue text.