apache / apache/lucene

Handle soft deletes via LiveDocsFormat [LUCENE-10392]

Open
#11,428 12 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We have been using doc values to handle soft deletes until now, but this is a bit of a hack as it:
 - forces users to reserve a field name for doc values
- generally doesn't read directly from doc values, instead docs values help populate bitsets and then reads are performed via these bitsets

It would also be more natural to have both hard and soft deletes handled by the same file format?

---
Migrated from [LUCENE-10392](https://issues.apache.org/jira/browse/LUCENE-10392) by Adrien Grand (@jpountz), updated May 17 2022

Contributor guide

Open the contributing guide

Research direction

Start by reading the LiveDocsFormat context and the issue's discussion of doc values, bitsets, and hard versus soft deletes. The issue does not name files or tests, and the desired implementation boundary and completion criteria need to be established before work begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.