Encode the characters as just Unicode points

Right now (for various reasons) each character is encoded as incrementing numbers. I don't think this is necessary with the new lib. Instead we could simplify some preprocessing by just encoding everything as the Unicode code point in the parallel text format files.

Then the charmaps can be built with simple bash commands (maybe some seq2seq scripts for this too).

The downside may be that will may want to encode full words rather than only characters (at least for dealing with common words).

Locate the preprocessing that assigns incrementing character numbers, the parallel text format files, and the charmaps or seq2seq scripts mentioned in the issue. Compare Unicode code-point encoding with the current representation and resolve whether common full-word encoding is also required. Done means the encoding choice is agreed and the affected preprocessing and charmap workflow are consistently updated.

Written by the indexing model from the issue text.