apache / apache/lucene

Query-time join collectors could maybe be more efficient [LUCENE-4771]

Open
#5,836 12 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:join type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I was looking `@ these` collectors on #5830 and I noticed:

- SingleValued collector (SV) pulls FieldCache.getTerms and adds the bytes to a bytesrefhash per-collect.
- MultiValued collector (MV) pulls FieldCache.getDocTermsOrds, but doesnt use the ords, just looks up each value and adds the bytes per-collect.

I think instead its worth investigating if SV should use getTermsIndex, and both collectors just collect-up their per-segment ords in something like a BitSet[maxOrd].

When asked for the terms at the end in getCollectorTerms(), they could merge these into one BytesRefHash.

Of course, if you are going to turn around and execute the query against the same searcher anyway (is this the typical case?), this could even be more efficient: No need to hash or instantiate all the terms in memory, we could do postpone the lookups to SeekingTermSetTermsEnum.accept()/nextSeekTerm() i think... somehow :)

---
Migrated from [LUCENE-4771](https://issues.apache.org/jira/browse/LUCENE-4771) by Robert Muir (@rmuir), updated Sep 17 2013
Attachments: [LUCENE-4771_prototype_without_bug.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4771/LUCENE-4771_prototype_without_bug.patch), [LUCENE-4771_prototype.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4771/LUCENE-4771_prototype.patch), [LUCENE-4771-prototype.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4771/LUCENE-4771-prototype.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading the SingleValued and MultiValued collectors, then trace FieldCache.getTerms, getDocTermsOrds, and getCollectorTerms(). Compare the proposed getTermsIndex and per-segment ordinal approach with the SeekingTermSetTermsEnum path. Done requires a decided and validated efficiency improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.