apache / apache/lucene

Lucene (unexpected ) fsync on existing segments [LUCENE-9889]

Open
#10,928 1 comment 0 reactions 0 assignees View on GitHub
affects-version:7.7.2 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If one of the existing segment files is opened by another (say a 3rd party) process, it can causing a parallel commit to fail with an error complaining about the index files to be locked by another process. Upon debugging, I see that fsync is being called during commit on already existing segment files, and failure to open the file in write mode causes this. But this should not be an expected behavior since there is no reason for a commit to open an existing segment file in WRITE mode to fsync. Please note that in this case, the index file was also a part of a saved commit point, so there is all the more reason to not fsync it.    
 
The line of code I am referring to is as below:
try (final FileChannel file = FileChannel.open(fileToSync, isDir ? StandardOpenOption.READ : StandardOpenOption.WRITE))
 
in method fsync(Path fileToSync, boolean isDir) of the class file
 
lucene\core\src\java\org\apache\lucene\util\IOUtils.java
 
 
Opening this Jira after discussion with Mike McCandless and Michael Sokolov on the dev mailing list here:
[Lucene - Java Developer - Lucene (unexpected ) fsync on existing segments (nabble.com)](https://lucene.472066.n3.nabble.com/Lucene-unexpected-fsync-on-existing-segments-td4469731.html)

---
Migrated from [LUCENE-9889](https://issues.apache.org/jira/browse/LUCENE-9889) by Rahul Goswami, updated May 04 2021

Contributor guide

Open the contributing guide

Research direction

Start with lucene/core/src/java/org/apache/lucene/util/IOUtils.java and the fsync(Path fileToSync, boolean isDir) method, then review the linked developer mailing-list discussion for context. Trace how commit handles existing segment files and verify that an existing saved-commit file is not opened in write mode for fsync; done means the reported parallel-commit failure is addressed with appropriate regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.