apache / apache/lucene

Avoid per-document int[] allocation in BytesRefHash#shrink() for term vectors path

Open
#16,507 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

When `storeTermVectors` is enabled, `BytesRefHash.clear()` is called after every document via `TermVectorsConsumerPerField.finishDocument()` -> `reset()` -> `bytesHash.clear(false)`. Inside `clear()`, `shrink(count)` may allocate a brand new `int[] ids` if the previous document had way more unique terms than the current one.

This means for workloads with var-len term-vectors-enabled fields, we can produce many short-lived `int[]` allocation per document. For example:
- Doc N has 40 unique terms in a term-vector-enabled field -> `ids[]` size is 128
- Doc N+1 has only 14 unique terms -> `shrink()` allocates a new `int[32]`, the old `int[128]` gets GCed

Image

Over millions of documents, this may create GC pressure.

Contributor guide

Open the contributing guide

Research direction

Start with BytesRefHash#clear() and shrink(), then trace the call path through TermVectorsConsumerPerField.finishDocument() and reset(). Confirm how varying per-document term counts replace the ids array, and verify that the term-vectors path no longer creates the described short-lived allocation while existing tests continue to pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.