apache / apache/lucene

BinaryDictionary may fail reading a file [LUCENE-5855]

Open
#6,917 1 comment 0 reactions 0 assignees View on GitHub
affects-version:4.9 legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The following code may fail on other JVMs, e.g. on Android:

final ByteBuffer tmpBuffer = ByteBuffer.allocateDirect(size);
final ReadableByteChannel channel = Channels.newChannel(dictIS);
final int read = channel.read(tmpBuffer);
if (read != size) {
throw new EOFException("Cannot read whole dictionary");
}

The code expects that channel.read() will fill entire buffer contents, which is however not the case: ReadableByteChannel.read() may even decide to read 0 bytes and this mimicks InputStream.read() behavior. The current Lucene code accidentally works on Oracle Java because Oracle Java channel.read() fills the buffer fully (just check the Channels.newChannel() implementation).

---
Migrated from [LUCENE-5855](https://issues.apache.org/jira/browse/LUCENE-5855) by Martin Vysny

Contributor guide

Open the contributing guide

Research direction

Start by locating the BinaryDictionary code described in the issue and read the surrounding channel and ByteBuffer handling. Check the ReadableByteChannel.read() contract across JVM implementations, including Android. Done means dictionary files are read reliably when a channel returns fewer bytes than requested, with the existing EOF behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.