Lucene index corrupted [LUCENE-6359]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
We've been using lucene to index logs of XML requestsusing custom tokenizer for months.
The XML log volume is 1-2Gb each day. The ./index currently is 2,3G
Today we've got a corrupted index:
Fri Mar 13 18:08:17 CET 2015:com.siap.WebServices.Utility.UtiIndexerLucene caught an exception: 1470975561 java.io.EOFException
> e.toString():java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/local/standard-atlante-plus/logs/WSAP/index/segments_za5l"),
> e.getMessage():read past EOF: MMapIndexInput(path="/usr/local/standard-atlante-plus/logs/WSAP/index/segments_za5l")
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:285)()
> org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)()
> org.apache.lucene.store.DataInput.readInt(DataInput.java:86)()
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272)()
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:76)()
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)()
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)()
> org.apache.lucene.index.IndexReader.open(IndexReader.java:256)()
> com.siap.WebServices.Utility.UtiIndexerLucene.search(UtiIndexerLucene.java
indexing:/usr/local/standard-atlante-plus/logs/WSAP/log_2015-03-13.20150313180041536-7138;20150313180041244-50344-0-20150313180041529-50346.log caught an exception: 1900866631 org.apache.lucene.index.CorruptIndexException
> > e.toString():org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file,
> > e.getMessage():failed to locate current segments_N file
> > org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:233)()
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1168)()
> > com.siap.WebServices.Utility.UtiIndexerLucene.indexFile(UtiIndexerLucene.java:417)(8c089dcb26049ca479266c90ff1a2c49)
> > com.siap.WebServices.Utility.UtiLogPrintingThread.run(UtiLogPrintingThread.java:146)(35e13d9710ff2dae1c1d582f4502b6a8)
> > Somma controllo versione:2cb199870d8058b03c4773affe406f9d
> > Server info:Apache Tomcat/5.5.26@0:0:0:0:0:0:0:1(siapvm3.gattinoni.it)
If you want the compressed ./index I've it
---
Migrated from [LUCENE-6359](https://issues.apache.org/jira/browse/LUCENE-6359) by Bartolomeo Nicolotti
Environment:
```
uname -a
Linux 2.6.32-358.23.2.el6.x86_64
$ java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)
```
Contributor guide
Research direction
Start with the reported stack traces in MMapDirectory.java, SegmentInfos.java, DirectoryReader.java, and IndexWriter.java, then inspect the failing segments_za5l and segments_N files if the compressed index is available. Reproduce the failure under the reported Java and Linux environment and determine what operation leaves the index unreadable. Done means the corruption cause is isolated and a reliable regression test or reproduction is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100