Make Tokenizers deliver their final offsets [LUCENE-5386]

Tokenizers _must_ have an implementation of #end() in which they set up the final offset. Currently, nothing enforces this. end() has a useful implementation in TokenStream, so just making it abstract is not attractive.

Proposal: add

abstract int finalOffset();

to tokenizer, and then make

void end() {
super.end();
int fo = finalOffset();
offsetAttr.setOffsets(fo, fo);
}

or something to that effect.

Other alternative to be considered depending on how this looks.

---
Migrated from [LUCENE-5386](https://issues.apache.org/jira/browse/LUCENE-5386) by Benson Margulies (@bimargulies-google), updated Jan 10 2014

Start with the Tokenizer and TokenStream entry points described in the issue, then inspect the existing end() implementations in tokenizer subclasses. Identify how final offsets are currently supplied and update the affected implementations so the tokenizer contract is enforced; done means tokenizers consistently deliver their final offsets without losing TokenStream's useful default behavior.

Written by the indexing model from the issue text.