apache / apache/lucene

Efficient way to calculate hardLiveDocs count of a SegmentReader when both hard and soft deletes are present

Open
#15,352 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

## Background

We are currently working on a feature in OpenSearch to support context aware segment within OpenSearch which involves maintaining multiple IndexWriter instances, one for each group, within a shard to collocate related data into same segment or group of segments. The design is detailed in the following RFCs and LLD:

* [OpenSearch RFC](https://github.com/opensearch-project/OpenSearch/issues/18576)
* [Lucene RFC](https://github.com/apache/lucene/issues/13387)
* [OpenSearch LLD](https://github.com/opensearch-project/OpenSearch/issues/19530)

## Current Use Case

With Context Aware Segment, within a shard, writes are routed to respective group-specific `IndexWriter` instances. To maintain consistent versioning across writers during update operation, we perform a **hard delete** of the previous document version in the parent (accumulating) `IndexWriter` whenever a new version is added to a group-specific writer.

## Problem Description

Currently with just soft deletes enabled, during OpenSearch's DocRep recovery, OpenSearch [uses `SegmentReader.hardLiveDocs` to query live docs](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/common/lucene/Lucene.java#L942) from segments with hard deletes (which may have gotten introduced due to IndexWriter hitting non-aborted exceptions). The number of liveDocs is efficiently derived as:

`segmentReader.maxDoc() - segmentReader.getSegmentInfo().getDelCount()`

However, by performing both soft and hard delete on a context aware enabled Lucene Index, the above calculation breaks down as `segmentReader.getSegmentInfo().getDelCount()` no longer provide the accurate live delete count on a segment. Based on [Lucene's unit tests for mixed deletes](https://github.com/apache/lucene/blob/f2da05b25396a72adb07895c8858a15841c3c6a9/lucene/core/src/test/org/apache/lucene/index/TestSoftDeletesRetentionMergePolicy.java#L696), the only reliable method to get the live doc count is to iterate through the hardLiveDocs and count the set bits.

## Performance Impact

This iterative counting operation is computationally expensive for large segments and can potentially cause significant performance regressions during shard recovery.

## Ask from this issue

Is there a more optimized, direct way to retrieve the count of live documents from a SegmentReader's hardLiveDocs when a segment has undergone both hard and soft deletes?

Contributor guide

Open the contributing guide

Research direction

Start with SegmentReader.hardLiveDocs and the linked OpenSearch recovery code in Lucene.java. Read the mixed-deletes cases in TestSoftDeletesRetentionMergePolicy.java, then investigate whether Lucene exposes a direct live-document count for this state. Done means an agreed optimized approach is identified and its correctness and recovery performance are covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.