apache / apache/lucene

Having a modular Doc Values format [LUCENE-9236]

Open
#10,276 9 comments 0 reactions 0 assignees View on GitHub
legacy-jira-label:docValues legacy-jira-priority:Minor module:core/index type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today DocValues Consumer/Producer require override 5 different methods, even if you only want to use one and given that one given field can only support one doc values type at same time.

 

In the attached PR I’ve implemented a new modular version of those classes (consumer/producer) each one having a single responsibility and writing in the same unique file.

This is mainly a refactor of the existing format opening the possibility to override or implement the sub-format you need.

 

I’ll do in 3 steps:
1. Create a CompositeDocValuesFormat and moving the code of Lucene80DocValuesFormat in separate classes, without modifying the inner code. At same time I created a Lucene85CompositeDocValuesFormat based on these changes.
1. I’ll introduce some basic components for writing doc values in general such as:
1. DocumentIdSetIterator Serializer: used in each type of field based on an IndexedDISI.
1. Document Ordinals Serializer: Used in Sorted and SortedSet for deduplicate values using a dictionary.
1. Document Boundaries Serializer (optional used only for multivalued fields: SortedNumeric and SortedSet)
1. TermsEnum Serializer: useful to write and read the terms dictionary for sorted and sorted set doc values.
1. I’ll create the new Sub-DocValues format using the previous components.

 

PR: 

---
Migrated from [LUCENE-9236](https://issues.apache.org/jira/browse/LUCENE-9236) by juan camilo rodriguez duran, updated Mar 10 2020
Sub-tasks:
- #10243

Contributor guide

Open the contributing guide

Research direction

Start by reading the DocValues Consumer and Producer APIs, then inspect Lucene80DocValuesFormat and the referenced PR 1282. Compare the proposed CompositeDocValuesFormat and Lucene85CompositeDocValuesFormat with the three planned component stages; done means the modular sub-formats preserve the existing format behavior while allowing individual components to be replaced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.