apache / apache/lucene

ExitableDirectoryReader should not check on every term [LUCENE-7324]

Open
#8,378 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I looked at ExitableDirectoryReader and to me checking the timeout on every term is a pretty heavy operation. I wonder if we can relax that a bit and begin with checking when we pull Terms and maybe only every Nth term by default? I wonder if we even can make it a function of the number of therms ie log(numTerms) I think it's pretty trappy to have something that won't perform well or has the risk to not scale well in lucene core?

---
Migrated from [LUCENE-7324](https://issues.apache.org/jira/browse/LUCENE-7324) by Simon Willnauer (@s1monw)

Contributor guide

Open the contributing guide

Research direction

Start by reading ExitableDirectoryReader and trace where timeout checks occur during term iteration. Measure the cost of checking every term and investigate a bounded or adaptive check interval; done means the checking frequency is reduced without losing the intended timeout behavior, with supporting validation added where the existing implementation allows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.