apache / apache/lucene

IndexWriter.getTragicException() may not reflect all corrupting exceptions (notably: NoSuchFileException) [LUCENE-8692]

Open
#9,738 17 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Backstory...

Solr has a "LeaderTragicEventTest" which uses MockDirectoryWrapper's `corruptFiles` to introduce corruption into the "leader" node's index and then assert that this solr node gives up it's leadership of the shard and another replica takes over.

This can currently fail sporadically (but usually reproducibly - see SOLR-13237) due to the leader not giving up it's leadership even after the corruption causes an update/commit to fail. Solr's leadership code makes this decision after encountering an exception from the IndexWriter based on wether `IndexWriter.getTragicException()` is (non-)null.

----
While investigating this, I created an isolated Lucene-Core equivilent test that demonstrates the same basic situation:
- Gradually cause corruption on an index untill (otherwise) valid execution of IW.add() + IW.commit() calls throw an exception to the IW client.
- assert that if an exception is thrown to the IW client, `getTragicException()` is now non-null.

It's fairly easy to make my new test fail reproducibly – in every situation I've seen the underlying exception is a `NoSuchFileException` (ie: the randomly introduced corruption was to delete some file).

---
Migrated from [LUCENE-8692](https://issues.apache.org/jira/browse/LUCENE-8692) by Chris M. Hostetter (@hossman), updated Mar 12 2019
Attachments: [LUCENE-8692_test.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8692/LUCENE-8692_test.patch), [LUCENE-8692.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8692/LUCENE-8692.patch) (versions: 3)
Linked issues:
- [SOLR-13237](https://issues.apache.org/jira/browse/SOLR-13237)

Contributor guide

Open the contributing guide

Research direction

Start with IndexWriter.getTragicException() and the attached LUCENE-8692 test patch, then inspect the MockDirectoryWrapper corruption scenario around IW.add() and IW.commit(). Reproduce the NoSuchFileException case and verify that an exception reaching the IndexWriter client is reflected by a non-null tragic exception, while checking the related LeaderTragicEventTest behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.