apache / apache/lucene

MergeScheduler should not implement Closeable [LUCENE-5885]

Open
#6,947 7 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

MergeScheduler implements Closeable and IndexWriter calls ms.close() when it's closed. But MergeScheduler can be shared between several writers, which means closing it by any particular writer is wrong. We should rather implement some ref-counting logic such that each IW will call incRef() in the ctor, and decRef() on close(), and MergeScheduler will truly close when the ref-count hits 0.

As it is now, if you share a MergeScheduler between writers and close() does something terminating, I doubt if it really works.

Also, when I look at ConcurrentMergeScheduler.close(), it calls sync() which joins all MergeThreads. But if that CMS instance is shared between few IWs, doesn't it mean that a single IW calling close() waits on MergeThreads that execute merges of other IWs!?!? This seems ... wrong?

---
Migrated from [LUCENE-5885](https://issues.apache.org/jira/browse/LUCENE-5885) by Shai Erera (@shaie)
Attachments: [LUCENE-5885.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5885/LUCENE-5885.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start by reading MergeScheduler, IndexWriter, and ConcurrentMergeScheduler, focusing on the close and sync interactions described in the issue. Review the attached LUCENE-5885.patch for prior context. Done means shared schedulers remain usable until their last writer closes, without one writer waiting on or terminating merges belonging to others.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.