apache / apache/parquet-java

Add ParquetJoiner feature

未关闭
#2,873 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Hadoop Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

**Overview**
ParquetJoiner feature is similar to ParquetRewrite class. ParquetRewrite allows to stitch files with the same schema into a single file while ParquetJoiner should enable stitching files with different schemas into a single file. That is possible when: 1) the number of rows in the main and extra files is the same, 2) the ordering of rows in the main and extra files is the same. Main benefit of ParquetJoiner is performance, for the cases when you join/stitch Terabytes/Petabytes of data that seemingly simple low level API can be up to 10x more resource efficient.

**Implementation details**
ParquetJoiner allows to specify the main input parquet file and extra input parquet files. ParquetJoiner will copy the main input as binary data and write extra input files with row groups adjusted to the main input. If main input is much larger than extra inputs then a lot of resources will be saved by working with the main input as binary.

**Use-case examples**
A very large Parquet based dataset(dozens or hundreds of fields/Terabytes of data daily/Petabytes of historical partitions). The task is to modify a column or add a new column to it for all the historic data. It is trivial using Spark, but taking into consideration the share scale of a dataset it will take a lot of resources to do that.

**Side notes**
Note that this class of problems could be in theory solved by storing main input and extra inputs in HMS/Iceberg bucketed tables and use a view that joins those tables on the fly into the final version but in practice there is often a requirement to merge parquet files and have a single parquet sources in the file system.

**Use-case implementation details using Apache Spark**
You can use Apache Spark to perform the join with ParquetJoiner, read the large main input and prepare the right side of a join in a way that each file on the left have a corresponding file on the right and it preserves records ordering on the right side in the same order as on the left side, that allows the whole input on the left and right to have the same number of files and the same number of records in corresponding files and the same ordering of records in each file pair. Then run ParquetJoiner in parallel for each file pair and perform a join. [Code sample to be provided separately?]

 

**Reporter**: [Maksim Konstantinov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maxnevemind)
#### PRs and other links:
- [GitHub Pull Request #1273](https://github.com/apache/parquet-mr/pull/1273)
- [GitHub Pull Request #1335](https://github.com/apache/parquet-mr/pull/1335)

**Note**: *This issue was originally created as [PARQUET-2430](https://issues.apache.org/jira/browse/PARQUET-2430). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先检查现有的 ParquetRewrite 类,以及 GitHub Pull Requests #1273 和 #1335 中讨论的实现。确认应如何组合一个主 Parquet 输入和行数及顺序匹配的额外输入,然后验证结果是否保留主输入,同时添加经过调整的行组和额外 schema。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
20/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。