apache / apache/parquet-java

Parquet Writer doesn't support to write list of Groups directly into hadoop

Ouverte
#2,331 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Component: Java Component: Parquet Priority: Blocker Type: bug
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

**Problem :-**  Parquet Writer doesn't support to write list of Groups directly into hadoop.

Cause of Problem :- Suitable classes are not available
Constructors are not visible

**All possible solution which i tried -**
- ParquetWriter
- ParquetFileWriter
- ExampleParquetWriter
- ParquetRecordWriter

**Gradle dependency**
> compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
 

**Is there any way to writer parquet file into Hadoop with GroupSupport?**

 

**Problem description**

I am trying to create the object of ParquetWriter class which accepts the argument (OutputFile, Mode, WriteSupport, CompressionCodecName, int, boolean, Configuration, int, ParquetProperties). But this constructor has default access modifier. I can't able to access it.

 I also used ParquetFileWriter class but it doesn't show any group support.

 

**ParquetWriter**

**![image-2019-05-31-10-41-45-674.png](https://issues.apache.org/jira/secure/attachment/12970426/image-2019-05-31-10-41-45-674.png)**

 **ParquetFileWriter**
```java

ParquetFileWriter writer = new ParquetFileWriter(HadoopOutputFile.fromPath(writePathFile, configuration), schema, Mode.CREATE, DEFAULT_BLOCK_SIZE, MAX_PADDING_SIZE_DEFAULT) ;
writer.start();
```
 

 It creates  to create parquet file in hadoop but can't facilitates to write  List of Groups in parquet file

 

**ExampleParquetWriter**
```java

ParquetWriter writer = ExampleParquetWriter.builder(writePathFile).withConf(configuration).withType(getSchema()).build();
System.out.println("Number of groups to write:" + groups.size());
for (Group g : groups) {
writer.write(g);

```
 Doesn't support for writing  parquet file in hadoop

 

 

 

 

**Reporter**: [Udit Joshi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=udit0612)
#### Original Issue Attachments:
- [image-2019-05-31-10-41-45-674.png](https://issues.apache.org/jira/secure/attachment/12970426/image-2019-05-31-10-41-45-674.png)

**Note**: *This issue was originally created as [PARQUET-1587](https://issues.apache.org/jira/browse/PARQUET-1587). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par examiner ParquetWriter, ParquetFileWriter, ExampleParquetWriter et ParquetRecordWriter dans l’API parquet-hadoop 1.10.1, ainsi que l’utilisation montrée de ExampleParquetWriter. Déterminez comment GroupSupport est censé fonctionner avec la sortie Hadoop et si le constructeur inaccessible fait partie du chemin nécessaire. C’est terminé lorsqu’une méthode prise en charge pour écrire une liste de Groups dans un fichier Parquet Hadoop est disponible et démontrée.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.