Merges should periodically check for abort [LUCENE-10658]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Rolling back an IndexWriter without committing shouldn't take long (i.e., less than several seconds), and Elasticsearch cluster coordination [relies on](https://github.com/elastic/elasticsearch/issues/88055) this assumption. If some merges are taking place, the rollback can take several minutes as merges only check for abort when writing to files via [MergeRateLimiter](https://github.com/apache/lucene/blob/3d7d85f245381f84c46c766119695a8645cde2b8/lucene/core/src/java/org/apache/lucene/index/MergeRateLimiter.java#L117-L119). Merging a completion field, for example, can take a long time without touching output files. Another reason merges should periodically check for abort is its outputs will be discarded.
---
Migrated from [LUCENE-10658](https://issues.apache.org/jira/browse/LUCENE-10658) by Nhat Nguyen (@dnhatn), updated Jul 28 2022
Contributor guide
Research direction
Start with IndexWriter rollback and MergeRateLimiter, especially the abort check referenced in lucene/core/src/main/java/org/apache/lucene/index/MergeRateLimiter.java. Reproduce a rollback while a merge such as a completion-field merge is running, then verify that merges periodically notice abort and rollback completes within several seconds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100