apache / apache/lucene

Store IndexedDISI in a separate file from doc values data

Open
#16,298 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Currently, for sparse fields (where not all documents have a value), we write two structures into the same .dvd file: the IndexedDISI presence bitmap followed by the packed values. IndexedDISI is a compressed list of docIds which helps in answering question like 'Does document X have a value for this field?'

We already separated out skip index into a different file - https://github.com/apache/lucene/pull/15976. I wonder if we should do the same for indexedDISI structure? As for example, when we are reading skip index, and we hit a block which is `YES_IF_PRESENT`(all values are in range but some docs are missing), we need to seek into .dvd file to find which docs are present([link](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/DocValuesRangeIterator.java#L354)). I think this shouldn't be required ie interleaving this metadata with actual values.

We can either separate out this into a new file or stuff it inside skip index. I prefer the former as it is much clean.

Contributor guide

Open the contributing guide

Research direction

Start with DocValuesRangeIterator.java around the linked lookup and inspect how sparse-field IndexedDISI and packed values are written into .dvd files. Compare the existing skip-index file separation in pull request 15976. Done means the IndexedDISI metadata is stored separately from doc values data so range iteration does not seek into the .dvd file for presence information.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.