derrickoswald / derrickoswald/CIMSpark

Kryo

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Spark recommends [Kryo serialization](https://github.com/EsotericSoftware/kryo).
This issue tracks tasks to ensure that this serialization framework is used and works correctly.
- find and fix cases where generated RDD are not using Kryo (e.g. MEMORY_ONLY)
- [implement](https://github.com/EsotericSoftware/kryo#implementing-a-serializer) built-in [Kryo serialization](https://stackoverflow.com/questions/36144618/spark-kryo-register-a-custom-serializer) to take advantage of the bitfields information that stores which fields are populated
- write and perform benchmark tests to check if it's really better

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the generated RDD paths and the MEMORY_ONLY cases mentioned in the issue. Then read the linked Kryo serialization guidance and identify how benchmark tests are organized. Done means the listed RDD cases use Kryo, built-in serialization handles the populated-field bitfields, and benchmark results show whether it improves performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.