apache / apache/parquet-java

Add ParquetJoiner feature

Ouverte
#2,873 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Component: Hadoop Component: Parquet Priority: Major Type: enhancement
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

**Overview**
ParquetJoiner feature is similar to ParquetRewrite class. ParquetRewrite allows to stitch files with the same schema into a single file while ParquetJoiner should enable stitching files with different schemas into a single file. That is possible when: 1) the number of rows in the main and extra files is the same, 2) the ordering of rows in the main and extra files is the same. Main benefit of ParquetJoiner is performance, for the cases when you join/stitch Terabytes/Petabytes of data that seemingly simple low level API can be up to 10x more resource efficient.

**Implementation details**
ParquetJoiner allows to specify the main input parquet file and extra input parquet files. ParquetJoiner will copy the main input as binary data and write extra input files with row groups adjusted to the main input. If main input is much larger than extra inputs then a lot of resources will be saved by working with the main input as binary.

**Use-case examples**
A very large Parquet based dataset(dozens or hundreds of fields/Terabytes of data daily/Petabytes of historical partitions). The task is to modify a column or add a new column to it for all the historic data. It is trivial using Spark, but taking into consideration the share scale of a dataset it will take a lot of resources to do that.

**Side notes**
Note that this class of problems could be in theory solved by storing main input and extra inputs in HMS/Iceberg bucketed tables and use a view that joins those tables on the fly into the final version but in practice there is often a requirement to merge parquet files and have a single parquet sources in the file system.

**Use-case implementation details using Apache Spark**
You can use Apache Spark to perform the join with ParquetJoiner, read the large main input and prepare the right side of a join in a way that each file on the left have a corresponding file on the right and it preserves records ordering on the right side in the same order as on the left side, that allows the whole input on the left and right to have the same number of files and the same number of records in corresponding files and the same ordering of records in each file pair. Then run ParquetJoiner in parallel for each file pair and perform a join. [Code sample to be provided separately?]

 

**Reporter**: [Maksim Konstantinov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maxnevemind)
#### PRs and other links:
- [GitHub Pull Request #1273](https://github.com/apache/parquet-mr/pull/1273)
- [GitHub Pull Request #1335](https://github.com/apache/parquet-mr/pull/1335)

**Note**: *This issue was originally created as [PARQUET-2430](https://issues.apache.org/jira/browse/PARQUET-2430). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par examiner la classe ParquetRewrite existante et l’implémentation abordée dans les GitHub Pull Requests #1273 et #1335. Confirmez comment une entrée Parquet principale et des entrées supplémentaires ayant le même nombre de lignes et le même ordre doivent être combinées, puis vérifiez que le résultat conserve l’entrée principale tout en ajoutant les schémas supplémentaires avec des groupes de lignes ajustés.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
20/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.