apache / apache/lucene

optimization: when sorting by field, if index has one segment and field values are not needed, do not load String[] into field cache [LUCENE-2335]

Open
#3,411 13 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-label:gsoc2014 legacy-jira-priority:Minor module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Spinoff from java-dev thread "Sorting with little memory: A suggestion", started by Toke Eskildsen.

When sorting by SortField.STRING we currently ask FieldCache for a StringIndex on that field.

This can consumes tons of RAM, when the values are mostly unique (eg a title field), as it populates both int[] ords as well as String[] values.

But, if the index is only one segment, and the search sets fillFields=false, we don't need the String[] values, just the int[] ords. If the app needs to show the fields it can pull them (for the 1 page) from stored fields.

This can be a potent optimization – alot of RAM saved – for optimized indexes.

When fixing this we must take care to share the int[] ords if some queries do fillFields=true and some =false... ie, FieldCache will be called twice and it should share the int[] ords across those invocations.

---
Migrated from [LUCENE-2335](https://issues.apache.org/jira/browse/LUCENE-2335) by Michael McCandless (@mikemccand), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start by tracing sorting with SortField.STRING through FieldCache, comparing the fillFields=false and fillFields=true paths for a single-segment index. Inspect how StringIndex data is cached and shared across repeated calls. Done means the no-field-values path avoids loading unnecessary String[] values while continuing to share int[] ords safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.