ExitableDirectoryReader should not check on every term [LUCENE-7324]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I looked at ExitableDirectoryReader and to me checking the timeout on every term is a pretty heavy operation. I wonder if we can relax that a bit and begin with checking when we pull Terms and maybe only every Nth term by default? I wonder if we even can make it a function of the number of therms ie log(numTerms) I think it's pretty trappy to have something that won't perform well or has the risk to not scale well in lucene core?
---
Migrated from [LUCENE-7324](https://issues.apache.org/jira/browse/LUCENE-7324) by Simon Willnauer (@s1monw)
Contributor guide
Research direction
Start by reading ExitableDirectoryReader and trace where timeout checks occur during term iteration. Measure the cost of checking every term and investigate a bounded or adaptive check interval; done means the checking frequency is reduced without losing the intended timeout behavior, with supporting validation added where the existing implementation allows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100