Avoid per-document int[] allocation in BytesRefHash#shrink() for term vectors path
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
When `storeTermVectors` is enabled, `BytesRefHash.clear()` is called after every document via `TermVectorsConsumerPerField.finishDocument()` -> `reset()` -> `bytesHash.clear(false)`. Inside `clear()`, `shrink(count)` may allocate a brand new `int[] ids` if the previous document had way more unique terms than the current one.
This means for workloads with var-len term-vectors-enabled fields, we can produce many short-lived `int[]` allocation per document. For example:
- Doc N has 40 unique terms in a term-vector-enabled field -> `ids[]` size is 128
- Doc N+1 has only 14 unique terms -> `shrink()` allocates a new `int[32]`, the old `int[128]` gets GCed
Over millions of documents, this may create GC pressure.
Contributor guide
Research direction
Start with BytesRefHash#clear() and shrink(), then trace the call path through TermVectorsConsumerPerField.finishDocument() and reset(). Confirm how varying per-document term counts replace the ids array, and verify that the term-vectors path no longer creates the described short-lived allocation while existing tests continue to pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100