TextIndexManager checksum failed on s390x [LUCENE-8567]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Hello,
we are using Apache Lucene within Jetbrain's Youtrack and have runtime problems with IBM's J9 VM version 8, but the same bug occurs when using openjdk on the same platform. Maybe there is a problem with the big-endian byte order of the z-platform?
```java
[2018-11-14 18:08:11,470] 18:08:11,462 ERROR [`@extIndexManager`] [TextIndexManager ] [] Failed to index 111 documents
[2018-11-14 18:08:11,471] org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=fa0a2c04 actual=a7c0aac0 (resource=BufferedChecksumIndexInput(ExodusIndexInput[_0_Lucene50_0.doc]))
[2018-11-14 18:08:11,471] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,472] at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:99) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,472] at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4997) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,473] at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:576) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,473] at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:515) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,473] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,474] at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:719) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,474] at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3602) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,474] at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3577) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,475] at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1035) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,475] at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1078) \~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
[2018-11-14 18:08:11,475] at jetbrains.youtrack.textindex.TextIndexManagerImpl$LuceneIndexingContext.close(TextIndexManagerImpl.java:843) \~[youtrack-text-search-2018.3.jar:?]
[2018-11-14 18:08:11,476] at jetbrains.youtrack.textindex.TextIndexManagerBase$BatchIndexingJob.execute(TextIndexManagerBase.java:995) [youtrack-text-search-2018.3.jar:?]
[2018-11-14 18:08:11,476] at jetbrains.exodus.core.execution.DecoratorJob.executeDecorated(DecoratorJob.java:72) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,476] at jetbrains.youtrack.textindex.settings.TransactionalTextIndexJob.access$000(TransactionalTextIndexJob.java:10) [youtrack-text-search-2018.3.jar:?]
[2018-11-14 18:08:11,477] at jetbrains.youtrack.textindex.settings.TransactionalTextIndexJob$1.invoke(TransactionalTextIndexJob.java:25) [youtrack-text-search-2018.3.jar:?]
[2018-11-14 18:08:11,477] at jetbrains.teamsys.dnq.runtime.txn._Txn.runReadonly(_Txn.java:351) [jetbrains.teamsys.dnq.runtime-3438.jar:?]
[2018-11-14 18:08:11,477] at jetbrains.youtrack.textindex.settings.TransactionalTextIndexJob.execute(TransactionalTextIndexJob.java:23) [youtrack-text-search-2018.3.jar:?]
[2018-11-14 18:08:11,477] at jetbrains.exodus.core.execution.Job.run(Job.java:99) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,478] at jetbrains.exodus.core.execution.ThreadJobProcessor.executeJob(ThreadJobProcessor.java:133) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,478] at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doExecuteJob(JobProcessorQueueAdapter.java:243) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,478] at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doTimedJobs(JobProcessorQueueAdapter.java:195) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,479] at jetbrains.exodus.core.execution.JobProcessorQueueAdapter.doJobs(JobProcessorQueueAdapter.java:163) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,479] at jetbrains.exodus.core.execution.ThreadJobProcessor.run(ThreadJobProcessor.java:111) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,479] at jetbrains.exodus.core.execution.ThreadJobProcessor$1.run(ThreadJobProcessor.java:43) [xodus-utils-1.2.3501.jar:1.2.3501]
[2018-11-14 18:08:11,479] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
---
Migrated from [LUCENE-8567](https://issues.apache.org/jira/browse/LUCENE-8567) by Stefan Franz
Environment:
```
SLES 12.3 on s390x with IBM J9 VM
```
Contributor guide
Research direction
Start with the failure path in org.apache.lucene.codecs.CodecUtil.checkFooter and org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write, using the s390x/OpenJDK or IBM J9 environment described in the report. Reproduce the checksum mismatch during IndexWriter shutdown and identify a regression test or other verification showing indexing completes without the reported CorruptIndexException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100