Standard/Classic/UAX tokenizers could be more ram efficient [LUCENE-6913]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
These tokenizers map codepoints to character classes with the following datastructure (loaded in clinit):
```
private static char [] zzUnpackCMap(String packed) {
char [] map = new char[0x110000];
```
This requires 2MB RAM for each tokenizer class (in trunk 6MB if all 3 classes are loaded, in branch_5x 10MB since there are 2 additional backwards compat classes).
On the other hand, none of our tokenizers actually use a huge number of character classes, so `char` is overkill: e.g. this map can safely be a byte [] and we can save half the memory. Perhaps it could make these tokenizers faster too.
---
Migrated from [LUCENE-6913](https://issues.apache.org/jira/browse/LUCENE-6913) by Robert Muir (@rmuir), updated Aug 22 2016
Attachments: [LUCENE-6913.not.a.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6913/LUCENE-6913.not.a.patch)
Contributor guide
Assessment
This issue has not been assessed yet.