Speed up soft delete
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Soft deletes consume a lot of CPU when flushing docvalue updates or calculating the `numsToDelete` in `SoftDeleteRetentionMergePolicy`. I was looking for some way to speed up these operations. The new `DocIdSetIterator#intoBitset` interface seems to provide a good approach, which is as follows:
- [x] https://github.com/apache/lucene/pull/14529.
- [x] https://github.com/apache/lucene/pull/14552
- [ ] https://github.com/apache/lucene/pull/14531.
Another optimization I'm looking for is to expose the fact that soft deleted fields always use a single value, so that we can avoid having to go through the calculations to calculate the min/max/gcd (idea initially raised in https://github.com/apache/lucene/pull/12557). My current idea of API designing is pretty simple, but I'm not sure if it's good.
```
public abstract class NumericDocValues extends DocValuesIterator {
/**
* If the impl knows all docs have the same value, return the value, otherwise null.
*/
public Long singleValue() {
return null;
}
...
}
```
Contributor guide
Research direction
Start by reading SoftDeleteRetentionMergePolicy, NumericDocValues, and the DocIdSetIterator#intoBitset interface to understand the proposed optimization points. Review the completed pull requests 14529 and 14552 and the unfinished pull request 14531 before evaluating the singleValue API idea. Done requires a settled approach for speeding up soft-delete processing, with the relevant implementation and tests identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100