apache / apache/parquet-java

Parquet proto writer de-nest Protobuf wrapper classes

Open
#2,336 1 comment 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Existing Parquet protobuf writer support preserves the structure of any Protobuf Message objects.  This works well in most cases. However, when dealing with [Protobuf wrapper messages](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto), users may prefer directly writing the de-nested value into the Parquet files, for ease of querying them directly (in query engine such as Hive/Presto). 

Proposal: 
- Implement a control flag, e.g., enableDenestingWrappers, to control whether or not to denest Protobuf wrapper classes. 
- When this flag is set to true, write the Protobuf wrapper classes as single primitive fields, based on the type of the wrapped **value** field.
 

|Protobuf Type|Parquet Type|
|
|-|-|-|
|BoolValue|boolean|
|
|BytesValue|binary|
|
|DoubleValue|double|
|
|FloatValue|float|
|
|Int32Value|int64 (32-bit, signed)|
|
|Int64Value|int64 (64-bit, signed)|
|
|StringValue|binary (string)|
|
|UInt32Value|int64 (32-bit, unsigned)|
|
|UInt64Value|int64 (64-bit, unsigned)|



 |

**Reporter**: [Ying Xu](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=yxu-apache)

**Note**: *This issue was originally created as [PARQUET-1595](https://issues.apache.org/jira/browse/PARQUET-1595). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.