apache / apache/lucene

Lucene index corrupted [LUCENE-6359]

Open
#7,420 0 comments 0 reactions 0 assignees View on GitHub
affects-version:3.6.1 legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We've been using lucene to index logs of XML requestsusing custom tokenizer for months.
The XML log volume is 1-2Gb each day. The ./index currently is 2,3G
Today we've got a corrupted index:

Fri Mar 13 18:08:17 CET 2015:com.siap.WebServices.Utility.UtiIndexerLucene caught an exception: 1470975561 java.io.EOFException
> e.toString():java.io.EOFException: read past EOF: MMapIndexInput(path="/usr/local/standard-atlante-plus/logs/WSAP/index/segments_za5l"),
> e.getMessage():read past EOF: MMapIndexInput(path="/usr/local/standard-atlante-plus/logs/WSAP/index/segments_za5l")
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:285)()
> org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)()
> org.apache.lucene.store.DataInput.readInt(DataInput.java:86)()
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272)()
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:76)()
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)()
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)()
> org.apache.lucene.index.IndexReader.open(IndexReader.java:256)()
> com.siap.WebServices.Utility.UtiIndexerLucene.search(UtiIndexerLucene.java

indexing:/usr/local/standard-atlante-plus/logs/WSAP/log_2015-03-13.20150313180041536-7138;20150313180041244-50344-0-20150313180041529-50346.log caught an exception: 1900866631 org.apache.lucene.index.CorruptIndexException
> > e.toString():org.apache.lucene.index.CorruptIndexException: failed to locate current segments_N file,
> > e.getMessage():failed to locate current segments_N file
> > org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:233)()
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1168)()
> > com.siap.WebServices.Utility.UtiIndexerLucene.indexFile(UtiIndexerLucene.java:417)(8c089dcb26049ca479266c90ff1a2c49)
> > com.siap.WebServices.Utility.UtiLogPrintingThread.run(UtiLogPrintingThread.java:146)(35e13d9710ff2dae1c1d582f4502b6a8)
> > Somma controllo versione:2cb199870d8058b03c4773affe406f9d
> > Server info:Apache Tomcat/5.5.26@0:0:0:0:0:0:0:1(siapvm3.gattinoni.it)

If you want the compressed ./index I've it

---
Migrated from [LUCENE-6359](https://issues.apache.org/jira/browse/LUCENE-6359) by Bartolomeo Nicolotti
Environment:
```
uname -a
Linux 2.6.32-358.23.2.el6.x86_64
$ java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

```

Contributor guide

Open the contributing guide

Research direction

Start with the reported stack traces in MMapDirectory.java, SegmentInfos.java, DirectoryReader.java, and IndexWriter.java, then inspect the failing segments_za5l and segments_N files if the compressed index is available. Reproduce the failure under the reported Java and Linux environment and determine what operation leaves the index unreadable. Done means the corruption cause is isolated and a reliable regression test or reproduction is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.