derrickoswald / derrickoswald/CIMSpark
Kryo
- Dominant language
- Scala
- Stars
- 19
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Spark recommends [Kryo serialization](https://github.com/EsotericSoftware/kryo).
This issue tracks tasks to ensure that this serialization framework is used and works correctly.
- find and fix cases where generated RDD are not using Kryo (e.g. MEMORY_ONLY)
- [implement](https://github.com/EsotericSoftware/kryo#implementing-a-serializer) built-in [Kryo serialization](https://stackoverflow.com/questions/36144618/spark-kryo-register-a-custom-serializer) to take advantage of the bitfields information that stores which fields are populated
- write and perform benchmark tests to check if it's really better
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the generated RDD paths and the MEMORY_ONLY cases mentioned in the issue. Then read the linked Kryo serialization guidance and identify how benchmark tests are organized. Done means the listed RDD cases use Kryo, built-in serialization handles the populated-field bitfields, and benchmark results show whether it improves performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100