apache / apache/lucene

IndexWriter.optimize() throws NullPointerException [LUCENE-2280]

Open
#3,356 19 comments 0 reactions 0 assignees View on GitHub
affects-version:2.3.2 legacy-jira-label:IndexWriter legacy-jira-label:NPE legacy-jira-label:optimize legacy-jira-priority:Major module:core/index type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I am using lucene 2.3.2 search APIs for my application, i am indexing 45GB database which creates approax 200MB index file, after finishing the indexing and while running optimize() i can see NullPointerExcception thrown in my log and index file is getting corrupted, log says

-----
Caused by:
java.lang.NullPointerException
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:49)
at org.apache.lucene.store.IndexOutput.writeBytes(IndexOutput.java:40)
at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:566)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:135)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3273)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2968)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240)

-----
and this is happening quite frequently, although I am not able to reproduce it on demand, I saw an issue logged which is some what related to mine issue (http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200809.mbox/%3C6E4A40DB-5EFC-42DA-A857-D59F4EC3496B@mikemccandless.com%3E) but the only difference here is I am not using Store.Compress for my fields, i am using Store.NO instead. please note that I am using IBM JRE for my application.

Is this an issue with lucene?, if yes it is fixed in which version?

---
Migrated from [LUCENE-2280](https://issues.apache.org/jira/browse/LUCENE-2280) by Ritesh Nigam, updated Nov 30 2013
Environment:
```
Win 2003, lucene version 2.3.2, IBM JRE 1.6
```

Attachments: [lucene.jar](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2280/lucene.jar), [lucene.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2280/lucene.zip), [LuceneUtils.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2280/LuceneUtils.zip)

Contributor guide

Open the contributing guide

Research direction

Start with the stack trace through BufferedIndexOutput, SegmentMerger.mergeNorms, and IndexWriter.optimize(), using the Lucene 2.3.2 environment details and attached archives. Compare the behavior with the related report cited in the issue and determine whether the failure can be reproduced. Done means optimize completes without the NullPointerException or index corruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.