IndexWriter.getTragicException() may not reflect all corrupting exceptions (notably: NoSuchFileException) [LUCENE-8692]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Backstory...
Solr has a "LeaderTragicEventTest" which uses MockDirectoryWrapper's `corruptFiles` to introduce corruption into the "leader" node's index and then assert that this solr node gives up it's leadership of the shard and another replica takes over.
This can currently fail sporadically (but usually reproducibly - see SOLR-13237) due to the leader not giving up it's leadership even after the corruption causes an update/commit to fail. Solr's leadership code makes this decision after encountering an exception from the IndexWriter based on wether `IndexWriter.getTragicException()` is (non-)null.
----
While investigating this, I created an isolated Lucene-Core equivilent test that demonstrates the same basic situation:
- Gradually cause corruption on an index untill (otherwise) valid execution of IW.add() + IW.commit() calls throw an exception to the IW client.
- assert that if an exception is thrown to the IW client, `getTragicException()` is now non-null.
It's fairly easy to make my new test fail reproducibly – in every situation I've seen the underlying exception is a `NoSuchFileException` (ie: the randomly introduced corruption was to delete some file).
---
Migrated from [LUCENE-8692](https://issues.apache.org/jira/browse/LUCENE-8692) by Chris M. Hostetter (@hossman), updated Mar 12 2019
Attachments: [LUCENE-8692_test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8692/LUCENE-8692_test.patch), [LUCENE-8692.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8692/LUCENE-8692.patch) (versions: 3)
Linked issues:
- [SOLR-13237](https://issues.apache.org/jira/browse/SOLR-13237)
Contributor guide
Research direction
Start with IndexWriter.getTragicException() and the attached LUCENE-8692 test patch, then inspect the MockDirectoryWrapper corruption scenario around IW.add() and IW.commit(). Reproduce the NoSuchFileException case and verify that an exception reaching the IndexWriter client is reflected by a non-null tragic exception, while checking the related LeaderTragicEventTest behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100