Concordance: have separate tokens for search and for display?

##### Text version

0.2.5

##### Orange version

3.7.dev

##### Expected behavior
Can browse by hashtags or emojis in Concordance.

##### Actual behavior
Since Concordance has internal preprocessing to display proper sentences, it cannot find Twitter-unique tokens, such as hashtags and emojis.

##### Steps to reproduce the behavior
Twitter - Preprocess Text (tweet tokenizer) - Word Cloud + Concordance. When selecting a hashtag in WC, it is not displayed in Conc.

##### Additional info (worksheets, data, screenshots, ...)
We need to discuss how to solve this.
- If we allow user-defined tokenization in Concordance, then it won't display logical sentences.
- If we override any tokenization with our internal one, the user can't find hashtags or emojis (as one would expect from tweet tokenizer).

Option: have an additional input for Preprocessor, which would override default tokenization.
Problem: Preprocess does not output Preprocessor.

Option 2: provide a checkbox in Concordance that would disable default preprocessing and take tokens from the input.
Problem: Concordance can now again display junk, but at least that's what the user consciously wanted...? 😁

Option 3: implement clever matching, partial matching? Also would we be able to display full text without resorting to tokens? How or what do you match then and how do you construct the view?

Start by reproducing the Twitter - Preprocess Text (tweet tokenizer) → Word Cloud → Concordance workflow described in the issue. Compare the proposed separate-token, override, and matching approaches, including whether full text can be displayed while preserving searchable hashtags and emojis. Done means a documented design and an implementation that supports the intended browsing behavior without losing readable sentences.

Written by the indexing model from the issue text.