apache / apache/lucene

Support Query Rewriting Caching [LUCENE-3056]

Open
#4,129 6 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Out of #4114, its become apparent that using a Visitor / Walker isn't right for caching the rewrites of Querys. Although we still intend to introduce the Query / Walker for advanced query transformations, rewriting still serves a purpose for very specific implementation detail writing. As such, it can be very expensive. So I think we should introduce first class support for rewrite caching. I also feel the key is to make the caching as transparent as possible, to reduce the strain on Query implementors.

The TermState idea gave me the idea of maybe making a RewriteState / RewriteCache / RewriteInterceptor, which would be consulted for rewritten Querys. It would then maintain an internal cache that it would check. If a value wasn't found, it'd then call Query#rewrite, and cache the result.

By having this external rewrite source, people could 'pre' rewrite Querys if they were particularly expensive but also common.

---
Migrated from [LUCENE-3056](https://issues.apache.org/jira/browse/LUCENE-3056) by Chris Male, updated May 09 2016
Attachments: [LUCENE-3056.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3056/LUCENE-3056.patch) (versions: 2)
Linked issues:
- #4114

Contributor guide

Open the contributing guide

Research direction

Start by reading Query#rewrite, the linked issue #4114, and the attached LUCENE-3056.patch; no source files or tests are named in the issue. Compare the proposed RewriteState, RewriteCache, or RewriteInterceptor approaches and determine how transparent caching should interact with Query implementors. Done means an agreed design and implementation for consulting and populating a rewrite cache without replacing the planned advanced Query/Walker transformations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.