apache / apache/lucene

Standard/Classic/UAX tokenizers could be more ram efficient [LUCENE-6913]

Open
#7,971 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

These tokenizers map codepoints to character classes with the following datastructure (loaded in clinit):

```
private static char [] zzUnpackCMap(String packed) {
char [] map = new char[0x110000];
```

This requires 2MB RAM for each tokenizer class (in trunk 6MB if all 3 classes are loaded, in branch_5x 10MB since there are 2 additional backwards compat classes).

On the other hand, none of our tokenizers actually use a huge number of character classes, so `char` is overkill: e.g. this map can safely be a byte [] and we can save half the memory. Perhaps it could make these tokenizers faster too.

---
Migrated from [LUCENE-6913](https://issues.apache.org/jira/browse/LUCENE-6913) by Robert Muir (@rmuir), updated Aug 22 2016
Attachments: [LUCENE-6913.not.a.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6913/LUCENE-6913.not.a.patch)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.