apache / apache/lucene

Explore a combined DocValues/PostingsFormat that shares a single terms dict [LUCENE-5832]

Open
#6,894 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

With #6881, which adds ordinals support to block tree terms dict, it's in theory possible to create a codec that uses a single shared terms dict for both Sorted/SetDocValues and postings as long as the app "promises" that the given field(s) will always be indexed as both S/SDV and inverted.

The codec should ideally throw an exc if that promise was broken, instead of adding incorrect values to the index.

This would save RAM because at search time only one shared "terms index" is loaded, vs two today. Of course the forward index for S/SDV (mapping doc -> term ords) must still be separately stored from the inverted postings (mapping term ord -> docs).

---
Migrated from [LUCENE-5832](https://issues.apache.org/jira/browse/LUCENE-5832) by Michael McCandless (@mikemccand), 2 votes

Contributor guide

Open the contributing guide

Research direction

Start by reviewing #6881 and the migrated LUCENE-5832 discussion to understand the existing block tree terms dictionary and ordinals support. Determine the codec entry points and tests involved before defining the shared terms dictionary design. Done means a codec can share one terms dictionary for postings and Sorted/SetDocValues while rejecting fields that violate the stated indexing promise.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.