apache / apache/lucene

waitForMerges deadlocks if background merge fails [LUCENE-3373]

Open
#4,446 9 comments 0 reactions 0 assignees View on GitHub
affects-version:3.0.3 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

waitForMerges can deadlock if a merge fails for ConcurrentMergeScheduler

this is because the merge thread will die, but pending merges are still available

normally, the merge thread will pick up the next merge once it finishes the previous merge, but in the event of a merge exception, the pending work is not resumed, but waitForMerges won't complete until all pending work is complete

i worked around this by overriding doMerge() like so:

```Java
protected final void doMerge(MergePolicy.OneMerge merge) throws IOException {
try {
super.doMerge(merge);
} catch (Throwable exc) {
// Just logging the exception and not rethrowing
// insert logging code here
}
}
```

Here's the rough steps i used to reproduce this issue:
override doMerge like so

```Java
protected final void doMerge(MergePolicy.OneMerge merge) throws IOException {
try {Thread.sleep(500L);} catch (InterruptedException e) { }
super.doMerge(merge);
throw new IOException("fail");
}
```

then, if you do the following:
loop 50 times:
addDocument // any doc
commit
waitForMerges // This will deadlock sometimes

[SOLR-2017](https://issues.apache.org/jira/browse/SOLR-2017) may be related to this (stack trace for deadlock looked related)

---
Migrated from [LUCENE-3373](https://issues.apache.org/jira/browse/LUCENE-3373) by Tim Smith, 2 votes, updated May 23 2022

Contributor guide

Open the contributing guide

Research direction

Start with ConcurrentMergeScheduler, its doMerge path, and waitForMerges; reproduce the issue using the supplied failing doMerge override, 50 addDocument/commit iterations, and waitForMerges. Trace what happens to pending merges after the merge thread throws, and verify that waitForMerges completes while pending work is resumed or reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.