Suggesters: allow to associate a user-specified key (int) with a string [LUCENE-4516]

As a user, I'd like to associate a “foreign key” with a string (rather: final node) in the suggester index (in addition to the rank). For example, I’d like to add “Lucene in Action” with key 1933988177 (the ISBN) and some rank to a WFST or AnalyzingSuggester. A completion would return the completed string and the key associated with each entry (i.e. final nodes get a “key” field (int), which is returned in the LookupResult). That foreign key could also be used for fast de-duping (no more string/byte array comparisons).

There may be workarounds for the “foreign key” use case –it seems that lots of data structures would be affected by storing a user-provided key with final nodes, which therefore may not be a viable path. It may be possible to encode the foreign key in the transducer’s output instead.

**Discussion on java-user@lucene:**

Mike McCandless:

This is maybe the same idea as
#5557 ? Could you simply stuff your ISBN onto the end of the suggestion (ie enroll Lucene in
Action|1933988177)?

Dawid Weiss:

Just remember that if your suffixes are unique then you'll be expanding the automaton quite a bit (unique suffix paths).

D.

Mike:

That's a good point... encoding into the FST's output may be better.

---
Migrated from [LUCENE-4516](https://issues.apache.org/jira/browse/LUCENE-4516) by Oliver Christ

Start by reading the WFST and AnalyzingSuggester implementations and the LookupResult type to map how final nodes, ranks, and returned suggestions are represented. Compare storing the user key with encoding it in the transducer output, then define the affected data structures and tests needed to verify returned keys and fast de-duplication.

Written by the indexing model from the issue text.