apache / apache/lucene

pendingNumDocs doesn't match totalMaxDoc if tragedy on flush() [LUCENE-9621]

Open
#10,661 3 comments 0 reactions 0 assignees View on GitHub
affects-version:8.6.3 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

While implementing a test to trigger an OutOfMemoryError on flush() in https://github.com/apache/lucene-solr/pull/2088, I noticed that the OOME was followed by an assertion failure on rollback with the following stacktrace:

```java
java.lang.AssertionError: pendingNumDocs 1 != 0 totalMaxDoc
at __randomizedtesting.SeedInfo.seed([ABBF17C4E0FCDEE5:DDC8E99910AFC8FF]:0)
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2398)
at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5196)
at org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5186)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3932)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3874)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3853)
at org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllRepeated(TestIndexWriterDelete.java:496)
```

We should probably look into how exactly we behave with this kind of tragedy on flush().

---
Migrated from [LUCENE-9621](https://issues.apache.org/jira/browse/LUCENE-9621) by Michael Froh (@msfroh), updated Dec 11 2020

Contributor guide

Open the contributing guide

Research direction

Start with IndexWriter.doFlush(), rollbackInternal(), and maybeCloseOnTragicEvent(), then inspect TestIndexWriterDelete.testDeleteAllRepeated() and the reported stack trace. Reproduce the OutOfMemoryError during flush and verify that rollback completes without the pendingNumDocs/totalMaxDoc assertion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.