apache / apache/lucene

Morelikethis queries are very slow compared to other search types [LUCENE-1690]

Open
#2,764 14 comments 0 reactions 0 assignees View on GitHub
affects-version:2.4.1 legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The MoreLikeThis object performs term frequency lookups for every query. From my testing that's what seems to take up the majority of time for MoreLikeThis searches.

For some (I'd venture many) applications it's not necessary for term statistics to be looked up every time. A fairly naive opt-in caching mechanism tied to the life of the MoreLikeThis object would allow applications to cache term statistics for the duration that suits them.

I've got this working in my test code. I'll put together a patch file when I get a minute. From my testing this can improve performance by a factor of around 10.

---
Migrated from [LUCENE-1690](https://issues.apache.org/jira/browse/LUCENE-1690) by Richard Marr, updated Apr 22 2013
Attachments: [LruCache.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1690/LruCache.patch), [LUCENE-1690.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1690/LUCENE-1690.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start with the MoreLikeThis object and review the attached LruCache.patch and LUCENE-1690.patch to understand the proposed term-frequency caching. Confirm how an opt-in cache should live for the MoreLikeThis object, then verify that MoreLikeThis queries avoid repeated lookups while preserving their results and improving the reported performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.