apache / apache/lucene

TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader failure due to DWPT race condition

Open
#13,352 0 comments 0 reactions 0 assignees View on GitHub
type:test
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This replicates rarely for me locally. To me this indicates a race condition instead of some logic error.

```
java.lang.AssertionError: seqNo=114262 vs maxSeqNo=114261
at __randomizedtesting.SeedInfo.seed([905AD19F00667602]:0)
at org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:567)
at org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:178)
at org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:155)
at org.apache.lucene.index.DocumentsWriterPerThread.finishDocuments(DocumentsWriterPerThread.java:328)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:273)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1558)
at org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1603)
at org.apache.lucene.tests.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:315)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:781)
at java.base/java.lang.Thread.run(Thread.java:1583)

```

This is very similar to: https://github.com/elastic/elasticsearch/issues/105122#issuecomment-2098412230

Either we are updating `maxSeqNo` incorrectly, or there is a race condition in DWPT that both tests have hit

### Gradle command to reproduce

```
./gradlew test --tests TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader -Dtests.seed=905AD19F00667602 -Dtests.locale=chr-Cher-US -Dtests.timezone=MST7MDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```

Contributor guide

Open the contributing guide

Research direction

Start with TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader and run the supplied Gradle command with its seed. Trace the reported sequence-number assertion through DocumentsWriterDeleteQueue, DocumentsWriterPerThread, DocumentsWriter, IndexWriter, and RandomIndexWriter. Done means the suspected DWPT race or maxSeqNo problem is understood and the reproducer no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.