FilteringTokenFilter should never corrupt the tokenstream graph [LUCENE-4065]

Currently removers like stopfilter have an option (true/false) to enable position increments.

If its true: it both inserts gaps where necessary AND propagates gaps down the stream.
If its false: it does neither, which can totally mess up the tokenstream graph (e.g. move synonyms to another word).

There are totally valid natural usecases for false, where you don't want gaps because you want phrasequeries to act as if the word was never actually there.

But 'not inserting gaps' is separate from proper propagation of existing gaps.

So I think we should provide an option (either fix 'false' or make it an enum), where you still get a legit tokenstream and dont totally screw it up, but you simply omit gaps.

See #4921 for more information (Where we at least fixed this case to not begin the tokenstream with posinc=0)

---
Migrated from [LUCENE-4065](https://issues.apache.org/jira/browse/LUCENE-4065) by Robert Muir (@rmuir), 4 votes, updated Feb 23 2018
Attachments: [LUCENE-4065_test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4065/LUCENE-4065_test.patch)
Linked issues:
- [SOLR-11968](https://issues.apache.org/jira/browse/SOLR-11968)
- #5706
- [SOLR-8089](https://issues.apache.org/jira/browse/SOLR-8089)

Start by reading the stopfilter behavior described in the issue and inspect the attached LUCENE-4065_test.patch alongside issue #4921. Verify how disabling position increments affects existing gaps and tokenstream graph positions, then confirm that the chosen behavior preserves graph validity while omitting newly inserted gaps.

Written by the indexing model from the issue text.