apache / apache/lucene

Speed up soft delete

Open
#14,521 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Soft deletes consume a lot of CPU when flushing docvalue updates or calculating the `numsToDelete` in `SoftDeleteRetentionMergePolicy`. I was looking for some way to speed up these operations. The new `DocIdSetIterator#intoBitset` interface seems to provide a good approach, which is as follows:

- [x] https://github.com/apache/lucene/pull/14529.
- [x] https://github.com/apache/lucene/pull/14552
- [ ] https://github.com/apache/lucene/pull/14531.

Another optimization I'm looking for is to expose the fact that soft deleted fields always use a single value, so that we can avoid having to go through the calculations to calculate the min/max/gcd (idea initially raised in https://github.com/apache/lucene/pull/12557). My current idea of API designing is pretty simple, but I'm not sure if it's good.

```
public abstract class NumericDocValues extends DocValuesIterator {

/**
* If the impl knows all docs have the same value, return the value, otherwise null.
*/
public Long singleValue() {
return null;
}

...

}
```

Contributor guide

Open the contributing guide

Research direction

Start by reading SoftDeleteRetentionMergePolicy, NumericDocValues, and the DocIdSetIterator#intoBitset interface to understand the proposed optimization points. Review the completed pull requests 14529 and 14552 and the unfinished pull request 14531 before evaluating the singleValue API idea. Done requires a settled approach for speeding up soft-delete processing, with the relevant implementation and tests identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.