apache / apache/lucene

Updatable OrdinalMap [LUCENE-6496]

Open
#7,555 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The MultiDocValues.OrdinalMap that we have to today requires a rebuild on each reopen. When the OrdinalMap has been built, lookups are fast and the logic is simple. Many time rebuilding the the OrdinalMap isn't even an issue, because for low to medium cardinality fields the rebuilding doesn't take that much time. The time required to build the OrdinalMap depends on the number of unique terms in a field.

For high cardinality fields (lets say >= 1M terms) rebuilding the OrdinalMap can take some time to complete. This can then impact the NRT aspect of many applications (facets may rely on ordinal maps to be rebuilt before a new search can happen after the reopen).

I like to explore a different OrdinalMap implementation that doesn't need to be rebuilt on each reopen. There are simple improvements that can made:
- Lets say docs have only been marked as deleted, then we basically reuse the OrdinalMap that has already been built.
- If no new terms have been introduced we can just add segment ordinal to global ordinal lookups to the OrdinalMap that has already been built.

I think a complete OrdinalMap rebuild is inevitable, but it would be great if we could rebuild on a flush / merge instead of on each reopen.

---
Migrated from [LUCENE-6496](https://issues.apache.org/jira/browse/LUCENE-6496) by Martijn van Groningen (@martijnvg), 1 vote
Attachments: [LUCENE-6496.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6496/LUCENE-6496.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start by locating the current MultiDocValues.OrdinalMap implementation and review the attached LUCENE-6496.patch. Trace how ordinal maps are rebuilt during reopen, flush, and merge, then define a design that preserves lookups while avoiding unnecessary reopen-time rebuilds. Done means the implementation and tests demonstrate reuse after deletions or when no new terms are introduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.