Fix analyzer bugs documented in TestRandomChains [LUCENE-4641]

TestRandomChains.java found a lot of bugs, some of which are hard to fix. So we blacklisted certain analysis components from the test.

But we really need to fix these, some of these bugs are bad, and they impact users with e.g. highlighting (SOLR-4137 and so on):
```
// TODO: fix those and remove
private static final Set> brokenComponents = Collections.newSetFromMap(new IdentityHashMap,Boolean>());
static {
// TODO: can we promote some of these to be only
// offsets offenders?
Collections.>addAll(brokenComponents,
// TODO: fix basetokenstreamtestcase not to trip because this one has no CharTermAtt
EmptyTokenizer.class,
// doesn't actual reset itself!
CachingTokenFilter.class,
// doesn't consume whole stream!
LimitTokenCountFilter.class,
// Not broken: we forcefully add this, so we shouldn't
// also randomly pick it:
ValidatingTokenFilter.class,
// NOTE: these by themselves won't cause any 'basic assertions' to fail.
// but see #4993, if any
// tokenfilter that combines words (e.g. shingles) comes after them,
// this will create bogus offsets because their 'offsets go backwards',
// causing shingle or whatever to make a single token with a
// startOffset thats > its endOffset
// (see #4812 for a list of other offenders here)
// broken!
NGramTokenizer.class,
// broken!
NGramTokenFilter.class,
// broken!
EdgeNGramTokenizer.class,
// broken!
EdgeNGramTokenFilter.class,
// broken!
WordDelimiterFilter.class,
// broken!
TrimFilter.class
);
}

// TODO: also fix these and remove (maybe):
// Classes that don't produce consistent graph offsets:
private static final Set> brokenOffsetsComponents = Collections.newSetFromMap(new IdentityHashMap,Boolean>());
static {
Collections.>addAll(brokenOffsetsComponents,
ReversePathHierarchyTokenizer.class,
PathHierarchyTokenizer.class,
HyphenationCompoundWordTokenFilter.class,
DictionaryCompoundWordTokenFilter.class,
// TODO: corrumpts graphs (offset consistency check):
PositionFilter.class,
// TODO: it seems to mess up offsets!?
WikipediaTokenizer.class,
// TODO: doesn't handle graph inputs
ThaiWordFilter.class,
// TODO: doesn't handle graph inputs
CJKBigramFilter.class,
// TODO: doesn't handle graph inputs (or even look at positionIncrement)
HyphenatedWordsFilter.class,
// #5137: only if you pass 'false' to enablePositionIncrements!
TypeTokenFilter.class,
// TODO: doesn't handle graph inputs
CommonGramsQueryFilter.class
);
}
```

---
Migrated from [LUCENE-4641](https://issues.apache.org/jira/browse/LUCENE-4641) by Robert Muir (@rmuir), 3 votes, updated Mar 19 2014
Attachments: [LUCENE-4641_tests.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4641/LUCENE-4641_tests.patch)
Linked issues:
- #6047
- #6048
- #6082
- #6175
- #5721
- #6027
- #5137
- #4993
- #4812
- #5293
- [SOLR-4137](https://issues.apache.org/jira/browse/SOLR-4137)
- #4980
- #5555
- #5732

Start with TestRandomChains.java and the attached LUCENE-4641_tests.patch, then review the blacklisted analyzer components and their linked issues. Determine which component bugs can be addressed independently, reproduce the relevant failures, and run the analyzer tests. Done means the fixed components no longer need blacklist entries and the randomized checks pass without offset or graph inconsistencies.

Written by the indexing model from the issue text.