Recent Java 9 commit breaks fsync on directory [LUCENE-6169]

I open this issue to keep track of the communication with Oracle and OpenJDK about this:

Basically, what happens: In #6650 we added support to FSDirectory to be able to sync on directory metadata changes (means the contents of the directory itsself). This is very important on Unix system (maybe also on Windows), because fsyncing a single file does not necessarily writes the directory's contents to disk. Lucene uses this for commits. We first do an atomic rename of the segments file (to make the commit public), but we have to be sure that the rename operation is written to disk. Because of that we must fsync the directory.

To enforce this with plain system calls (libc), you open a directory for read and then call fsync. In java this can be done by opening a FileChannel on the direczory(for read) and call fc.force() on it.

Unfortunately the commit http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/e5b66323ae45 in OpenJDK 9 break this. The corresponding issue is https://bugs.openjdk.java.net/browse/JDK-8066915. The JDK now explicitly checks if a file is a directory and disallows opening a FileChannel on it. This breaks our commit safety.

Because this behaviour is undocumented (not even POSIX has explicit semantics for syncing directories), we know that it worked at least on MacOSX and Linux. The code in IOUtils is currently written in a way that it tries to sync the diretory, but swallows any Exception. So this change does not break Liucene, but it breaks our commit safety. During testing we assert that the fsync actually works on Linux and MacOSX, in production code the user will notice nothing.

We should take action and contact Alan Bateman about his commit and this issue on the mailing list, possibly through Rory O'Donnell.

---
Migrated from [LUCENE-6169](https://issues.apache.org/jira/browse/LUCENE-6169) by Uwe Schindler (@uschindler), 1 vote, updated Nov 22 2015
Linked issues:
- #6650

Start by reading the FSDirectory directory-sync work in #6650 and the current IOUtils handling described here. Review the linked OpenJDK commit and JDK-8066915, then follow the Lucene/OpenJDK communication; done means an agreed resolution for preserving or documenting directory fsync commit safety.

Written by the indexing model from the issue text.