IndexWriter race condition [LUCENE-7499]

separated out from #3659

I've seen race condition diagnostics for Lucene code that could be related to this. Line numbers are for Lucene 5.5.2, but the problem seems present in master too.

AFAICT: StandardDirectoryReader has an IndexWriter, and it checks things are up-to-date by checking if there are pending in-memory updates by looking at IndexWriter.SegmentInfos.version. That read is not synchronized relative to its write from IndexWriter.updateDocument

I'm not enough of a lucene expert to suggest how to fix this in detail. If I would guess at how Lucene roughly works, I would suggest the following

- when applying pending in-memory changes, this takes a lock, and under lock there is a "changed = true" assignment.
- when checking for changes, take lock, return changed.
- when writing pending in-memory changes, take lock, write out file, set "changed=false", release lock.

The problem can be reproduced by running some of the Gerrit acceptance tests under Threadsanitizer. I don't know of open-source tooling to set that up, unfortunately.

WARNING: ThreadSanitizer: data race (pid=21636)
Write of size 8 at 0x7f24ad9d3210 by thread T36 (mutexes: write M431922409782456832):
`#0` org.apache.lucene.index.SegmentInfos.changed()V (SegmentInfos.java:944)
`#1` org.apache.lucene.index.IndexWriter.newSegmentName()Ljava/lang/String; (IndexWriter.java:1652)
`#2` org.apache.lucene.index.DocumentsWriter.ensureInitialized(Lorg/apache/lucene/index/DocumentsWriterPerThreadPool$ThreadState;)V (DocumentsWriter.java:391)
`#3` org.apache.lucene.index.DocumentsWriter.updateDocument(Ljava/lang/Iterable;Lorg/apache/lucene/analysis/Analyzer;Lorg/apache/lucene/index/Term;)Z (DocumentsWriter.java:445)
`#4` org.apache.lucene.index.IndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (IndexWriter.java:1477)
`#5` com.google.gerrit.lucene.AutoCommitWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (AutoCommitWriter.java:100)
`#6` org.apache.lucene.index.TrackingIndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)J (TrackingIndexWriter.java:55)
`#7` com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Long; (AbstractLuceneIndex.java:250)
`#8` com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Object; (AbstractLuceneIndex.java:247)
`#9` com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly()V (TrustedListenableFutureTask.java:108)
`#10` com.google.common.util.concurrent.InterruptibleTask.run()V (InterruptibleTask.java:41)
`#11` com.google.common.util.concurrent.TrustedListenableFutureTask.run()V (TrustedListenableFutureTask.java:77)
`#12` java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1142)
`#13` java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:617)
`#14` java.lang.Thread.run()V (Thread.java:745)
`#15` (Generated Stub)

Previous read of size 8 at 0x7f24ad9d3210 by thread T29 (mutexes: write M1060737507754061632):
`#0` org.apache.lucene.index.IndexWriter.nrtIsCurrent(Lorg/apache/lucene/index/SegmentInfos;)Z (IndexWriter.java:4592)
`#1` org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:282)
`#2` org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:261)
`#3` org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged()Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:251)
`#4` org.apache.lucene.index.DirectoryReader.openIfChanged(Lorg/apache/lucene/index/DirectoryReader;)Lorg/apache/lucene/index/DirectoryReader; (DirectoryReader.java:137)
`#5` com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Lorg/apache/lucene/search/IndexSearcher;)Lorg/apache/lucene/search/IndexSearcher; (WrappableSearcherManager.java:148)
`#6` com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Ljava/lang/Object;)Ljava/lang/Object; (WrappableSearcherManager.java:68)
`#7` org.apache.lucene.search.ReferenceManager.doMaybeRefresh()V (ReferenceManager.java:176)
`#8` org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking()V (ReferenceManager.java:253)
`#9` org.apache.lucene.search.ControlledRealTimeReopenThread.run()V (ControlledRealTimeReopenThread.java:245)
`#10` (Generated Stub)

---
Migrated from [LUCENE-7499](https://issues.apache.org/jira/browse/LUCENE-7499) by Han-Wen NIenhuys

Start by reading IndexWriter.nrtIsCurrent, SegmentInfos.changed, and StandardDirectoryReader.doOpenFromWriter at the Lucene 5.5.2 line references. Reproduce the reported race with the Gerrit acceptance tests under ThreadSanitizer, then trace the synchronization around updateDocument and refresh. Done means the concurrent read and write are safely coordinated and the race no longer appears.

Written by the indexing model from the issue text.