When loading FieldCache terms index, make terms data optional [LUCENE-2483]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Spinoff of #3455.
Now, when you load the terms index (FC.getTermsIndex), it loads two
arrays, ord (maps docID -> ord) and lookup (maps ord -> term).
But sometimes you don't need the lookup map (and, it's often very
costly in RAM usage, much moreso than the ord map).
EG if your index is a single segment, and your app doesn't need the
values (#3411).
Or, if you use a sort comparator that resolves ord -> term and v/v (eg
using terms dict).
So we should make it optional...
Also, similarly, we could merge getTerms/getTermsIndex. It's
dangerous today if you load terms and then termsIndex because you're
wasting tons of RAM; it'd be nicer if we could have a single cache
entry that'd "upgrade" itself to be an index (have the ords). This
single entry could then serve ords, ords+terms, or just terms.
---
Migrated from [LUCENE-2483](https://issues.apache.org/jira/browse/LUCENE-2483) by Michael McCandless (@mikemccand), updated May 09 2016
Contributor guide
Research direction
Start at FieldCache.getTermsIndex and trace how the ord and lookup arrays are loaded. Then review getTerms and getTermsIndex together; done means lookup data can be omitted when unnecessary and one cache entry can support terms, ords, or both without redundant memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100