fix or document termsIndexInterval/Divisor for 4.0 [LUCENE-4089]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
There are a few parameters on IndexWriterConfig/DirectoryReader that are going to be confusing unless we do something about it: at least documentation at the minimum:
- IWC.termsIndexInterval: really a codec parameter, actually ignored by 4.0's default impl (BlockTree)
- IWC.readerDivisor/DirectoryReader.divisor: really two things, if its -1 it means "don't load terms index", and this is respected by the current impls. Otherwise, it means "sample the terms index", and this is also actually ignored by 4.0's default impl (BlockTree)
I think people will be confused if they set these things and they do nothing. As far as fixing, I took a stab at this and its an annoyingly big change. But this is the rough sketch of one idea i had so far:
- remove interval: its only applicable if you customize codec and select a different terms index/dict impl anyway, so you can just pass this to FixedGap or whatever yourself.
- divisor: generalize this into something simple like a Map<String,String> of codec "parameters" that you set on IWC/IR. split divisor from "don't load terms index". define these as constants where they belong. I got unhappy here in the "splitting" part because I wanted the divisor part in TermsIndexReaderBase, but that doesnt extend FieldsProducer (where i wanted the "don't load" part) and wrap the terms dict, instead its backwards and terms dict wraps the TermsIndexReaderBase... maybe we should fix that too? I think this confusing the way it is but I didnt look at how difficult this would be.
---
Migrated from [LUCENE-4089](https://issues.apache.org/jira/browse/LUCENE-4089) by Robert Muir (@rmuir), updated May 09 2016
Linked issues:
- #5592
Contributor guide
Research direction
Start with IndexWriterConfig and DirectoryReader, then inspect the 4.0 default BlockTree implementation and the terms-index handling described in the issue. Decide whether accurate documentation alone resolves the confusion around termsIndexInterval, readerDivisor, and DirectoryReader.divisor; done means the supported semantics and ignored settings are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100