apache / apache/lucene

Add serialization support for RAMDirectory [LUCENE-3516]

Open
#4,590 0 comments 0 reactions 0 assignees View on GitHub
affects-version:3.4 legacy-jira-priority:Major module:core/store type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We are building Lucene indexes via Hadoop, and using byte[] for of RAMDirectory as intermediate storage. (we are using Hadoop's indexing contrib package)
Currently Java serialization is used, and seemed wasteful and not portable across languages.
Since RAMDirectory essentially is a collection of byte[], writing a serializer seemed easy.
Attached please find a utility class that does such. This includes a light performance comparison test comparing the java serialization:

Results: (input, number of files and avg file size, output, size percentage: our ser/java ser, time, serialization: percentage, our ser/java ser, deserialization: percentage, our ser/java ser

test 1:(3, 1k)
size: 66.93%
ser time: 1.89%
deser time: 4.48%

test 2: (100,100)
size: 95.16%
ser time: 4.01%
deser time: 13.36%

test 3: (3,50k)
size: 98.42%
ser time: 3.09%
deser time: 8.10%

test 4: (1,1k)
size: 41.70%
ser time: 1.85%
dser time: 3.85%

The implementation is very elementary, yet still much better than java ser. Esp on the time (avg 50x faster)

---
Migrated from [LUCENE-3516](https://issues.apache.org/jira/browse/LUCENE-3516) by John Wang
Attachments: [RAMDirectorySerializer.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3516/RAMDirectorySerializer.java)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached RAMDirectorySerializer.java and the existing RAMDirectory handling in Lucene. Check the included performance comparison test and determine how serialization should integrate with the project. Done means RAMDirectory contents can be serialized and restored without relying on Java serialization, with the reported comparison covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.