apache / apache/lucene

Improve AlreadyClosedException logging [LUCENE-10005]

Open
#11,044 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If we close the IndexWriter because of a tragic exception and later try to write into the index we got AlreadyClosedException exception like this:
```
2021-05-17 11:35:04,963 ERROR (qtp626742236-67894)----x:ct_returns_shard8_replica_n28-o.a.s.s.HttpSolrCall: null:org.apache.solr.common.SolrException: Error handling 'MERGEINDEXES' action
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:365)
...
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:671)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:685)
...
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1877842388-10.181.61.226-1618590425462:blk_1075816117_2075498 file=/solr/ct_returns/core_node31/data/index/_1z_Lucene50_0.doc
at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:880)
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:863)
at
```

This is rather misleading because 2021-05-17 is the date of the AlreadyClosedException, but the tragic exception (BlockMissingException in this case) could be much earlier. This is the exception we saved when we had to close the indexwriter.
 
We should emphasise that the tragic exception is not a fresh exception, but an exception we saved earlier.

---
Migrated from [LUCENE-10005](https://issues.apache.org/jira/browse/LUCENE-10005) by Andras Salamon (@asalamon74), updated Jun 17 2021

Contributor guide

Open the contributing guide

Research direction

Start by tracing IndexWriter.ensureOpen and the path that saves the tragic exception when the IndexWriter closes. Reproduce or inspect the reported AlreadyClosedException sequence, then verify that logging distinguishes the later closed-writer error from the earlier saved BlockMissingException. Done means the log makes clear that the tragic exception was saved earlier rather than freshly raised.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.