apache / apache/lucene

TestFieldsReader fails in 4.x with OOM [LUCENE-5066]

Open
#6,130 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Its FaultyIndexInput is broken (doesn't implement seek/clone correctly).

This causes it to read bogus data and try to allocate an enormous byte[] for a term.

The bug was previously hidden:
FaultyDirectory doesnt override openSlice, so CFS must not be used at flush if you want to trigger the bug.
FailtyIndexInput's clone is broken, it uses "new" but doesn't seek the clone to the right place. This causes a disaster with BufferedIndexInput (which it extends), because BufferedIndexInput (not just the delegate) must "know" its position since it has seek-within-block etc code...

It seems with this test (very simple one), that only 3.x codec triggers it because its term dict relies upon clone()'s being seek'd to right place.

I'm not sure what other codecs rely upon this, but imo we should also add a low-level test for directories that does something like this to ensure its really tested:

```Java
dir.createOutput(x);
dir.openInput(x);
input.seek(somewhere);
clone = input.clone();
assertEquals(somewhere, clone.getFilePointer());
```

---
Migrated from [LUCENE-5066](https://issues.apache.org/jira/browse/LUCENE-5066) by Robert Muir (@rmuir)
Attachments: [LUCENE-5066.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5066/LUCENE-5066.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading FaultyIndexInput and FaultyDirectory, then compare their clone, seek, and openSlice behavior with BufferedIndexInput. Reproduce the simple codec test that triggers the bogus term data and OOM. Done means cloned inputs preserve the file pointer and a low-level directory test covers this behavior without the allocation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.