optimization: when sorting by field, if index has one segment and field values are not needed, do not load String[] into field cache [LUCENE-2335]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Spinoff from java-dev thread "Sorting with little memory: A suggestion", started by Toke Eskildsen.
When sorting by SortField.STRING we currently ask FieldCache for a StringIndex on that field.
This can consumes tons of RAM, when the values are mostly unique (eg a title field), as it populates both int[] ords as well as String[] values.
But, if the index is only one segment, and the search sets fillFields=false, we don't need the String[] values, just the int[] ords. If the app needs to show the fields it can pull them (for the 1 page) from stored fields.
This can be a potent optimization – alot of RAM saved – for optimized indexes.
When fixing this we must take care to share the int[] ords if some queries do fillFields=true and some =false... ie, FieldCache will be called twice and it should share the int[] ords across those invocations.
---
Migrated from [LUCENE-2335](https://issues.apache.org/jira/browse/LUCENE-2335) by Michael McCandless (@mikemccand), updated May 09 2016
Contributor guide
Research direction
Start by tracing sorting with SortField.STRING through FieldCache, comparing the fillFields=false and fillFields=true paths for a single-segment index. Inspect how StringIndex data is cached and shared across repeated calls. Done means the no-field-values path avoids loading unnecessary String[] values while continuing to share int[] ords safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100