Ability to mlock certain fields from the terms dict [LUCENE-3214]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This is a hacked up prototype! It works but I'm not sure how to get this to a committable point.
The patch invokes mlock() (tested only on Linux), locking pages from the terms dictionary file that hold terms for a specified field. You can only do this with MMapDirectory.
I used this to lock pages for the "id" field in the NRT stress test; it's an alternative to MemoryCodec. But, it requires you set up the OS to allow the app/user to lock pages in RAM.
It works very well in reducing the NRT reopen latency even when large merges are running...
---
Migrated from [LUCENE-3214](https://issues.apache.org/jira/browse/LUCENE-3214) by Michael McCandless (@mikemccand)
Attachments: [LUCENE-3214.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3214/LUCENE-3214.patch)
Contributor guide
Research direction
Start by reviewing the attached LUCENE-3214.patch and the MMapDirectory and terms-dictionary paths it changes. Check the Linux-only mlock behavior, the required OS permissions, and the NRT stress-test use case; done means the prototype has a committable design with coverage for locking selected field pages and documented limitations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100