apache / apache/lucene

ByteBlockPool should allow different block sizes [LUCENE-2706]

Open
#3,780 0 comments 0 reactions 1 assignee Claimed by @s1monw View on GitHub
affects-version:4.0-ALPHA legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Currently ByteBlockPool has a fixed byteblock size of 1 << 15 bytes. This limit also makes BytesRefHash bound to 32kb buffers which is not ideal in many cases. For instance in MTQ (#4034) we might want to have a smaller block size of 1024 or 2048 for more efficiency. While this example suffers from another limitation since the blockSize used in TermsHashPerField is 32kb which in turn defines the size limit for a single term in the index. To make effective use of different byte block sizes we need to find a solution for this problem too but I think index statistics are on the way and max term length for a field could also make merges more efficient if we can alter the byte block size especially for fields with very short terms.

---
Migrated from [LUCENE-2706](https://issues.apache.org/jira/browse/LUCENE-2706) by Simon Willnauer (@s1monw)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.