apache / apache/lucene

When loading FieldCache terms index, make terms data optional [LUCENE-2483]

Open
#3,557 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Spinoff of #3455.

Now, when you load the terms index (FC.getTermsIndex), it loads two
arrays, ord (maps docID -> ord) and lookup (maps ord -> term).

But sometimes you don't need the lookup map (and, it's often very
costly in RAM usage, much moreso than the ord map).

EG if your index is a single segment, and your app doesn't need the
values (#3411).

Or, if you use a sort comparator that resolves ord -> term and v/v (eg
using terms dict).

So we should make it optional...

Also, similarly, we could merge getTerms/getTermsIndex. It's
dangerous today if you load terms and then termsIndex because you're
wasting tons of RAM; it'd be nicer if we could have a single cache
entry that'd "upgrade" itself to be an index (have the ords). This
single entry could then serve ords, ords+terms, or just terms.

---
Migrated from [LUCENE-2483](https://issues.apache.org/jira/browse/LUCENE-2483) by Michael McCandless (@mikemccand), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start at FieldCache.getTermsIndex and trace how the ord and lookup arrays are loaded. Then review getTerms and getTermsIndex together; done means lookup data can be omitted when unnecessary and one cache entry can support terms, ords, or both without redundant memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.