apache / apache/lucene

ConcurrentMergeScheduler/maxMergeCount default is too low [LUCENE-5705]

Open
#6,767 21 comments 0 reactions 0 assignees View on GitHub
affects-version:4.8 legacy-jira-fix-version:4.9 legacy-jira-priority:Minor module:core/other type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The default value for maxMergeCount in ConcurrentMergeScheduler is 2. This causes problems for Solr's dataimport handler when very large imports are done from a JDBC source.

What happens is that when three merge tiers are scheduled at the same time, the add/update thread will stop for several minutes while the largest merge finishes. In the meantime, the dataimporter JDBC connection to the database will time out, and when the add/update thread resumes, the import will fail because the ResultSet throws an exception. Setting maxMergeCount to 6 eliminates this issue for virtually any size import – although it is theoretically possible to have that many simultaneous merge tiers, I've never seen it.

As long as maxThreads is properly set (the default value of 1 is appropriate for most installations), I cannot think of a really good reason that the default for maxMergeCount should be so low. If someone does need to strictly control the number of threads that get created, they can reduce the number. Perhaps someone with more experience knows of a really good reason to make this default low?

I'm not sure what the new default number should be, but I'd like to avoid bikeshedding. I don't think it should be Integer.MAX_VALUE.

---
Migrated from [LUCENE-5705](https://issues.apache.org/jira/browse/LUCENE-5705) by Shawn Heisey (@elyograg), updated May 26 2014
Attachments: [dih-example.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5705/dih-example.patch), [infostream-s0build-shard.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5705/infostream-s0build-shard.zip), [LUCENE-5705.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5705/LUCENE-5705.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start with ConcurrentMergeScheduler and the LUCENE-5705 discussion and attached patches; determine whether a new maxMergeCount default has been agreed and how the reported Solr JDBC import failure is reproduced. Done requires a settled default value and validation that the merge behavior no longer causes the described import timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.