Add serialization support for RAMDirectory [LUCENE-3516]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
We are building Lucene indexes via Hadoop, and using byte[] for of RAMDirectory as intermediate storage. (we are using Hadoop's indexing contrib package)
Currently Java serialization is used, and seemed wasteful and not portable across languages.
Since RAMDirectory essentially is a collection of byte[], writing a serializer seemed easy.
Attached please find a utility class that does such. This includes a light performance comparison test comparing the java serialization:
Results: (input, number of files and avg file size, output, size percentage: our ser/java ser, time, serialization: percentage, our ser/java ser, deserialization: percentage, our ser/java ser
test 1:(3, 1k)
size: 66.93%
ser time: 1.89%
deser time: 4.48%
test 2: (100,100)
size: 95.16%
ser time: 4.01%
deser time: 13.36%
test 3: (3,50k)
size: 98.42%
ser time: 3.09%
deser time: 8.10%
test 4: (1,1k)
size: 41.70%
ser time: 1.85%
dser time: 3.85%
The implementation is very elementary, yet still much better than java ser. Esp on the time (avg 50x faster)
---
Migrated from [LUCENE-3516](https://issues.apache.org/jira/browse/LUCENE-3516) by John Wang
Attachments: [RAMDirectorySerializer.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3516/RAMDirectorySerializer.java)
Contributor guide
Research direction
Start by reviewing the attached RAMDirectorySerializer.java and the existing RAMDirectory handling in Lucene. Check the included performance comparison test and determine how serialization should integrate with the project. Done means RAMDirectory contents can be serialized and restored without relying on Java serialization, with the reported comparison covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100