inconsistency of tokenstream.end() with OffsetLimitTokenFilter and LimitTokenCountFilter [LUCENE-3088]

In #4137, we added some state and checks to MockTokenizer to validate that consumers
are properly using the tokenstream workflow (described here: http://lucene.apache.org/java/3_0_3/api/core/org/apache/lucene/analysis/TokenStream.html)

One inconsistency is the following steps:
4. The consumer calls incrementToken() until it returns false consuming the attributes after each call.
5. The consumer calls end() so that any end-of-stream operations can be performed.

In the case of these limitingfilters, end() is called on the Tokenizer **before** incrementToken() returns false. This is a little strange for a few reasons: one is that the tokenizer might not even be "ready" for end(), e.g. it might be coded where end() only works correctly if its entirely consumed. The other problem of course is that the finalOffset, the general use of end(), will most often be wrong in this case, so multi-valued field highlighting will not work.

We should probably figure out a way to address the inconsistency, some ideas are:
1. fixing the javadocs, perhaps documenting that end() could be called at any time, and accepting the fact that the finalOffset will be wrong.
1. the limiting filters could consume the rest of the tokens in a while (incrementToken()) loop to ensure totally proper behavior.
1. the limiting filters could do something tricky like override end() so that its not invoked on the Tokenizer in a surprising state. This is still evil but perhaps less evil than calling it "out of order".
1. ...

---
Migrated from [LUCENE-3088](https://issues.apache.org/jira/browse/LUCENE-3088) by Robert Muir (@rmuir)

The entry points named are OffsetLimitTokenFilter, LimitTokenCountFilter, MockTokenizer, and TokenStream.end()/incrementToken(), with the TokenStream workflow Javadoc as context. Start by tracing when each limiting filter calls end() and decide the intended ordering and finalOffset behavior. Done requires an agreed resolution plus regression coverage, but the issue does not name a test or file.

Written by the indexing model from the issue text.