apache / apache/parquet-java

Make MessageType serializable

Open
#2,035 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Background: I'm writing `ParquetRDD` implementation to run on Apache Spark. One of the features I want to provide is the ability to create instance of `ParqeutRDD` with a projection (specify columns to be loaded)

Problem: Schema information is not serializable. Neither `MessageType` nor any other class that extends `Type` extends `Serializable`.

Question: Is there any specific reason why not to have it the `extends Serializable`?
If not, then I will be more then happy to provide PR.

**Reporter**: [Pawel Szulc](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=rabbitonweb)

**Note**: *This issue was originally created as [PARQUET-850](https://issues.apache.org/jira/browse/PARQUET-850). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting MessageType and the Type class hierarchy to determine which schema classes need serialization support, then look for existing serialization tests in the repository. Done means the relevant schema objects can be serialized for ParquetRDD use without breaking existing behavior, with tests covering the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.