apache / apache/lucene

fix or document termsIndexInterval/Divisor for 4.0 [LUCENE-4089]

Open
#5,161 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

There are a few parameters on IndexWriterConfig/DirectoryReader that are going to be confusing unless we do something about it: at least documentation at the minimum:

- IWC.termsIndexInterval: really a codec parameter, actually ignored by 4.0's default impl (BlockTree)
- IWC.readerDivisor/DirectoryReader.divisor: really two things, if its -1 it means "don't load terms index", and this is respected by the current impls. Otherwise, it means "sample the terms index", and this is also actually ignored by 4.0's default impl (BlockTree)

I think people will be confused if they set these things and they do nothing. As far as fixing, I took a stab at this and its an annoyingly big change. But this is the rough sketch of one idea i had so far:
- remove interval: its only applicable if you customize codec and select a different terms index/dict impl anyway, so you can just pass this to FixedGap or whatever yourself.
- divisor: generalize this into something simple like a Map<String,String> of codec "parameters" that you set on IWC/IR. split divisor from "don't load terms index". define these as constants where they belong. I got unhappy here in the "splitting" part because I wanted the divisor part in TermsIndexReaderBase, but that doesnt extend FieldsProducer (where i wanted the "don't load" part) and wrap the terms dict, instead its backwards and terms dict wraps the TermsIndexReaderBase... maybe we should fix that too? I think this confusing the way it is but I didnt look at how difficult this would be.

---
Migrated from [LUCENE-4089](https://issues.apache.org/jira/browse/LUCENE-4089) by Robert Muir (@rmuir), updated May 09 2016
Linked issues:
- #5592

Contributor guide

Open the contributing guide

Research direction

Start with IndexWriterConfig and DirectoryReader, then inspect the 4.0 default BlockTree implementation and the terms-index handling described in the issue. Decide whether accurate documentation alone resolves the confusion around termsIndexInterval, readerDivisor, and DirectoryReader.divisor; done means the supported semantics and ignored settings are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.