Parquet proto writer de-nest Protobuf wrapper classes
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Existing Parquet protobuf writer support preserves the structure of any Protobuf Message objects. This works well in most cases. However, when dealing with [Protobuf wrapper messages](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto), users may prefer directly writing the de-nested value into the Parquet files, for ease of querying them directly (in query engine such as Hive/Presto).
Proposal:
- Implement a control flag, e.g., enableDenestingWrappers, to control whether or not to denest Protobuf wrapper classes.
- When this flag is set to true, write the Protobuf wrapper classes as single primitive fields, based on the type of the wrapped **value** field.
|Protobuf Type|Parquet Type|
|
|-|-|-|
|BoolValue|boolean|
|
|BytesValue|binary|
|
|DoubleValue|double|
|
|FloatValue|float|
|
|Int32Value|int64 (32-bit, signed)|
|
|Int64Value|int64 (64-bit, signed)|
|
|StringValue|binary (string)|
|
|UInt32Value|int64 (32-bit, unsigned)|
|
|UInt64Value|int64 (64-bit, unsigned)|
|
**Reporter**: [Ying Xu](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=yxu-apache)
**Note**: *This issue was originally created as [PARQUET-1595](https://issues.apache.org/jira/browse/PARQUET-1595). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing Parquet protobuf writer support and how it maps Protobuf Message objects to Parquet fields. Define the control flag for wrapper denesting and compare its output against the listed wrapper-to-primitive mappings. Done means the flag controls whether wrappers remain nested or are written as single primitive fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100