Store IndexedDISI in a separate file from doc values data
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Currently, for sparse fields (where not all documents have a value), we write two structures into the same .dvd file: the IndexedDISI presence bitmap followed by the packed values. IndexedDISI is a compressed list of docIds which helps in answering question like 'Does document X have a value for this field?'
We already separated out skip index into a different file - https://github.com/apache/lucene/pull/15976. I wonder if we should do the same for indexedDISI structure? As for example, when we are reading skip index, and we hit a block which is `YES_IF_PRESENT`(all values are in range but some docs are missing), we need to seek into .dvd file to find which docs are present([link](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/DocValuesRangeIterator.java#L354)). I think this shouldn't be required ie interleaving this metadata with actual values.
We can either separate out this into a new file or stuff it inside skip index. I prefer the former as it is much clean.
Contributor guide
Research direction
Start with DocValuesRangeIterator.java around the linked lookup and inspect how sparse-field IndexedDISI and packed values are written into .dvd files. Compare the existing skip-index file separation in pull request 15976. Done means the IndexedDISI metadata is stored separately from doc values data so range iteration does not seek into the .dvd file for presence information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100