apache / apache/lucene

NRTCachingDir can't handle large files [LUCENE-4484]

Open
#5,550 10 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I dug into this OOME, which easily repros for me on rev 1398268:
```
ant test -Dtestcase=Test4GBStoredFields -Dtests.method=test -Dtests.seed=2D89DD229CD304F5 -Dtests.multiplier=3 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ru -Dtests.timezone=Asia/Vladivostok -Dtests.file.encoding=UTF-8 -Dtests.verbose=true
```

The problem is the test got NRTCachingDir ... which cannot handle large files because it decides up front (when createOutput is called) whether the file will be in RAMDir vs wrapped dir ... so if that file turns out to be immense (which this test does since stored fields files can grow arbitrarily huge w/o any flush happening) then it takes unbounded RAM.

---
Migrated from [LUCENE-4484](https://issues.apache.org/jira/browse/LUCENE-4484) by Michael McCandless (@mikemccand), updated Oct 17 2012

Contributor guide

Open the contributing guide

Research direction

Reproduce with Test4GBStoredFields using the ant command and parameters in the issue. Read NRTCachingDir, especially createOutput and its RAMDir-versus-wrapped-directory decision, then trace how the stored-fields file grows without a flush. Done means the large-file test no longer causes unbounded RAM use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.