Unicode compression [LUCENE-1799]

In lucene-1793, there is the off-topic suggestion to provide compression of Unicode data. The motivation was a custom encoding in a Russian analyzer. The original supposition was that it provided a more compact index.

This led to the comment that a different or compressed encoding would be a generally useful feature.

BOCU-1 was suggested as a possibility. This is a patented algorithm by IBM with an implementation in ICU. If Lucene provide it's own implementation a freely avIlable, royalty-free license would need to be obtained.

SCSU is another Unicode compression algorithm that could be used.

An advantage of these methods is that they work on the whole of Unicode. If that is not needed an encoding such as iso8859-1 (or whatever covers the input) could be used.

---
Migrated from [LUCENE-1799](https://issues.apache.org/jira/browse/LUCENE-1799) by DM Smith, 2 votes, updated Nov 30 2013
Attachments: [Benchmark.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1799/Benchmark.java) (versions: 3), [LUCENE-1779.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1799/LUCENE-1779.patch), [LUCENE-1799_big.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1799/LUCENE-1799_big.patch), [LUCENE-1799.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1799/LUCENE-1799.patch) (versions: 14)

Start by reviewing the attached Benchmark.java and the LUCENE-1799 patch files, then read the issue discussion about BOCU-1, SCSU, and alternative encodings. The issue does not define a selected algorithm, implementation location, licensing decision, or testable completion condition.

Written by the indexing model from the issue text.