apache / apache/lucene

boolean docid set iterator improvement [LUCENE-1632]

Open
#2,706 3 comments 0 reactions 0 assignees View on GitHub
affects-version:2.4 legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This was first brought up in Lucene-1345. But Lucene-1345 conversation has digressed. As per suggested, creating a separate issue to track.
Added perf comparisons with boolean set iterators with current scorers
See patch

System: Ubunto,
java version "1.6.0_11"
Intel core2 Duo 2.44ghz

new milliseconds=470
new milliseconds=534
new milliseconds=450
new milliseconds=443
new milliseconds=444
new milliseconds=445
new milliseconds=449
new milliseconds=441
new milliseconds=444
new milliseconds=445
new total milliseconds=4565
old milliseconds=529
old milliseconds=491
old milliseconds=428
old milliseconds=549
old milliseconds=427
old milliseconds=424
old milliseconds=420
old milliseconds=424
old milliseconds=423
old milliseconds=422
old total milliseconds=4537

New/Old Time 4565/4537 (100.61715%)
OrDocIdSetIterator milliseconds=1138
OrDocIdSetIterator milliseconds=1106
OrDocIdSetIterator milliseconds=1065
OrDocIdSetIterator milliseconds=1066
OrDocIdSetIterator milliseconds=1065
OrDocIdSetIterator milliseconds=1067
OrDocIdSetIterator milliseconds=1072
OrDocIdSetIterator milliseconds=1118
OrDocIdSetIterator milliseconds=1065
OrDocIdSetIterator milliseconds=1069
OrDocIdSetIterator total milliseconds=10831
DisjunctionMaxScorer milliseconds=1914
DisjunctionMaxScorer milliseconds=1981
DisjunctionMaxScorer milliseconds=1861
DisjunctionMaxScorer milliseconds=1893
DisjunctionMaxScorer milliseconds=1886
DisjunctionMaxScorer milliseconds=1885
DisjunctionMaxScorer milliseconds=1887
DisjunctionMaxScorer milliseconds=1889
DisjunctionMaxScorer milliseconds=1891
DisjunctionMaxScorer milliseconds=1888
DisjunctionMaxScorer total milliseconds=18975
Or/DisjunctionMax Time 10831/18975 (57.080368%)
OrDocIdSetIterator milliseconds=1079
OrDocIdSetIterator milliseconds=1075
OrDocIdSetIterator milliseconds=1076
OrDocIdSetIterator milliseconds=1093
OrDocIdSetIterator milliseconds=1077
OrDocIdSetIterator milliseconds=1074
OrDocIdSetIterator milliseconds=1078
OrDocIdSetIterator milliseconds=1075
OrDocIdSetIterator milliseconds=1074
OrDocIdSetIterator milliseconds=1074
OrDocIdSetIterator total milliseconds=10775
DisjunctionSumScorer milliseconds=1398
DisjunctionSumScorer milliseconds=1322
DisjunctionSumScorer milliseconds=1320
DisjunctionSumScorer milliseconds=1305
DisjunctionSumScorer milliseconds=1304
DisjunctionSumScorer milliseconds=1301
DisjunctionSumScorer milliseconds=1304
DisjunctionSumScorer milliseconds=1300
DisjunctionSumScorer milliseconds=1301
DisjunctionSumScorer milliseconds=1317
DisjunctionSumScorer total milliseconds=13172
Or/DisjunctionSum Time 10775/13172 (81.80231%)
AndDocIdSetIterator milliseconds=330
AndDocIdSetIterator milliseconds=336
AndDocIdSetIterator milliseconds=298
AndDocIdSetIterator milliseconds=299
AndDocIdSetIterator milliseconds=310
AndDocIdSetIterator milliseconds=298
AndDocIdSetIterator milliseconds=298
AndDocIdSetIterator milliseconds=334
AndDocIdSetIterator milliseconds=298
AndDocIdSetIterator milliseconds=299
AndDocIdSetIterator total milliseconds=3100
ConjunctionScorer milliseconds=332
ConjunctionScorer milliseconds=307
ConjunctionScorer milliseconds=302
ConjunctionScorer milliseconds=350
ConjunctionScorer milliseconds=300
ConjunctionScorer milliseconds=304
ConjunctionScorer milliseconds=305
ConjunctionScorer milliseconds=303
ConjunctionScorer milliseconds=303
ConjunctionScorer milliseconds=299
ConjunctionScorer total milliseconds=3105
And/Conjunction Time 3100/3105 (99.83897%)

main contributors to the patch: Anmol Bhasin & Yasuhiro Matsuda

---
Migrated from [LUCENE-1632](https://issues.apache.org/jira/browse/LUCENE-1632) by John Wang, updated Nov 30 2013
Attachments: [Lucene-1632-patch.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1632/Lucene-1632-patch.txt)

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached Lucene-1632-patch.txt and tracing the current boolean set iterators alongside OrDocIdSetIterator, AndDocIdSetIterator, and the referenced scorers. Reproduce the reported iterator and scorer performance comparisons, then establish which benchmark results or behavior demonstrate that the improvement is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.