NRTCachingDir can't handle large files [LUCENE-4484]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I dug into this OOME, which easily repros for me on rev 1398268:
```
ant test -Dtestcase=Test4GBStoredFields -Dtests.method=test -Dtests.seed=2D89DD229CD304F5 -Dtests.multiplier=3 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ru -Dtests.timezone=Asia/Vladivostok -Dtests.file.encoding=UTF-8 -Dtests.verbose=true
```
The problem is the test got NRTCachingDir ... which cannot handle large files because it decides up front (when createOutput is called) whether the file will be in RAMDir vs wrapped dir ... so if that file turns out to be immense (which this test does since stored fields files can grow arbitrarily huge w/o any flush happening) then it takes unbounded RAM.
---
Migrated from [LUCENE-4484](https://issues.apache.org/jira/browse/LUCENE-4484) by Michael McCandless (@mikemccand), updated Oct 17 2012
Contributor guide
Research direction
Reproduce with Test4GBStoredFields using the ant command and parameters in the issue. Read NRTCachingDir, especially createOutput and its RAMDir-versus-wrapped-directory decision, then trace how the stored-fields file grows without a flush. Done means the large-file test no longer causes unbounded RAM use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100