WordDelimiterTokenFilter does not generate all tokens appropriately [LUCENE-8181]

When using word delimiter token filter some expected tokens are not generated.

When I try to analyze the text "ElasticSearch.TestProject"

I expect the tokens elastic, search, test, project, elasticsearch, testproject, elasticsearch.testproject to be generated since I have split_on_case_change, split_on_numerics on and using a whitespace tokenizer and have preserve original true.

But Actually I only see following tokens -
elasticsearch.testproject, elastic, search, test, project

---
Migrated from [LUCENE-8181](https://issues.apache.org/jira/browse/LUCENE-8181) by Atul, 1 vote, updated Mar 12 2018
Environment:
```
*Steps to reproduce*:
*1. Create index*
_PUT testindex_
_{_
_"settings" : {_
_"index" : {_
_"number_of_shards" : 2,_
_"number_of_replicas" : 2_
_},_
_"analysis": {_
_"filter": {_
_"wordDelimiter": {_
_"type": "word_delimiter",_
_"generate_word_parts": "true",_
_"generate_number_parts": "true",_
_"catenate_words": "false",_
_"catenate_numbers": "false",_
_"catenate_all": "false",_
_"split_on_case_change": "true",_
_"preserve_original": "true",_
_"split_on_numerics": "true",_
_"stem_english_possessive": "true"_
_}_
_},_
_"analyzer": {_
_"content_analyzer": {_
_"type": "custom",_
_"tokenizer": "whitespace",_
_"filter": [_
_"asciifolding",_
_"wordDelimiter",_
_"lowercase"_
_]_
_}_
_}_
_}_
_}_
_}_

*2. Analyze Text-*
_POST testindex/_analyze_
_{_
_"analyzer": "content_analyzer",_
_"text": "ElasticSearch.TestProject"_
_}_

*Following tokens are generated-*

{
"token": "elasticsearch-testproject",
"start_offset": 0,
"end_offset": 25,
"type": "word",
"position": 0
}
,
{
"token": "elastic",
"start_offset": 0,
"end_offset": 7,
"type": "word",
"position": 0
}
,
{
"token": "search",
"start_offset": 7,
"end_offset": 13,
"type": "word",
"position": 1
}
,
{
"token": "test",
"start_offset": 14,
"end_offset": 18,
"type": "word",
"position": 2
}
,
{
"token": "project",
"start_offset": 18,
"end_offset": 25,
"type": "word",
"position": 3
}

*Expected Result:*
Besides the above tokens even elasticsearch and testproject should be generated. such that the phrase query "elasticsearch testproject" should also match.

*Another example could be-*
Text *"Super-Duper-0-AutoCoder"* with above analyzer generates a token *autocoder* while the text *"Super-Duper-AutoCoder"* does NOT generate the token *autocoder*.
```

Start with the WordDelimiterTokenFilter behavior described in the issue and reproduce it using the whitespace tokenizer and the shown word_delimiter settings on "ElasticSearch.TestProject". Compare the generated tokens with the expected elasticsearch and testproject terms, then verify the second Super-Duper example; done means both cases generate the expected tokens without losing the existing output.

Written by the indexing model from the issue text.