apache / apache/lucene

Merges should periodically check for abort [LUCENE-10658]

Open
#11,694 4 comments 0 reactions 0 assignees View on GitHub
affects-version:9.3 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Rolling back an IndexWriter without committing shouldn't take long (i.e., less than several seconds), and Elasticsearch cluster coordination [relies on](https://github.com/elastic/elasticsearch/issues/88055) this assumption. If some merges are taking place, the rollback can take several minutes as merges only check for abort when writing to files via [MergeRateLimiter](https://github.com/apache/lucene/blob/3d7d85f245381f84c46c766119695a8645cde2b8/lucene/core/src/java/org/apache/lucene/index/MergeRateLimiter.java#L117-L119). Merging a completion field, for example, can take a long time without touching output files. Another reason merges should periodically check for abort is its outputs will be discarded.

---
Migrated from [LUCENE-10658](https://issues.apache.org/jira/browse/LUCENE-10658) by Nhat Nguyen (@dnhatn), updated Jul 28 2022

Contributor guide

Open the contributing guide

Research direction

Start with IndexWriter rollback and MergeRateLimiter, especially the abort check referenced in lucene/core/src/main/java/org/apache/lucene/index/MergeRateLimiter.java. Reproduce a rollback while a merge such as a completion-field merge is running, then verify that merges periodically notice abort and rollback completes within several seconds.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.