apache / apache/lucene

Complete overhaul of FieldCache API/Implementation [LUCENE-831]

Open
#1,906 162 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Motivation:
1) Complete overhaul the API/implementation of "FieldCache" type things...
a) eliminate global static map keyed on IndexReader (thus
eliminating synch block between completley independent IndexReaders)
b) allow more customization of cache management (ie: use
expiration/replacement strategies, disk backed caches, etc)
c) allow people to define custom cache data logic (ie: custom
parsers, complex datatypes, etc... anything tied to a reader)
d) allow people to inspect what's in a cache (list of CacheKeys) for
an IndexReader so a new IndexReader can be likewise warmed.
e) Lend support for smarter cache management if/when
IndexReader.reopen is added (merging of cached data from subReaders).
2) Provide backwards compatibility to support existing FieldCache API with
the new implementation, so there is no redundent caching as client code
migrades to new API.

---
Migrated from [LUCENE-831](https://issues.apache.org/jira/browse/LUCENE-831) by Chris M. Hostetter (@hossman), 5 votes, updated May 09 2016
Attachments: [ExtendedDocument.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/ExtendedDocument.java), [fieldcache-overhaul.032208.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/fieldcache-overhaul.032208.diff), [fieldcache-overhaul.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/fieldcache-overhaul.diff) (versions: 2), [LUCENE-831.03.28.2008.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/LUCENE-831.03.28.2008.diff), [LUCENE-831.03.30.2008.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/LUCENE-831.03.30.2008.diff), [LUCENE-831.03.31.2008.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/LUCENE-831.03.31.2008.diff), [LUCENE-831.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/LUCENE-831.patch) (versions: 14), [LUCENE-831-trieimpl.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-831/LUCENE-831-trieimpl.patch)
Linked issues:
- #2308
- #3739
- #2446
- #1818
- #4463
- #3209
- #2656

Contributor guide

Open the contributing guide

Research direction

Start by reading the FieldCache overhaul requirements and the attached fieldcache-overhaul.diff, LUCENE-831.patch, and related issue links. Trace the existing FieldCache API and implementation before comparing the proposed designs. Done means the API and implementation meet the listed cache-management and customization goals while preserving backward compatibility without redundant caching.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.