Parquet Writer doesn't support to write list of Groups directly into hadoop
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
**Problem :-** Parquet Writer doesn't support to write list of Groups directly into hadoop.
Cause of Problem :- Suitable classes are not available
Constructors are not visible
**All possible solution which i tried -**
- ParquetWriter
- ParquetFileWriter
- ExampleParquetWriter
- ParquetRecordWriter
**Gradle dependency**
> compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
**Is there any way to writer parquet file into Hadoop with GroupSupport?**
**Problem description**
I am trying to create the object of ParquetWriter class which accepts the argument (OutputFile, Mode, WriteSupport, CompressionCodecName, int, boolean, Configuration, int, ParquetProperties). But this constructor has default access modifier. I can't able to access it.
I also used ParquetFileWriter class but it doesn't show any group support.
**ParquetWriter**
****
**ParquetFileWriter**
```java
ParquetFileWriter writer = new ParquetFileWriter(HadoopOutputFile.fromPath(writePathFile, configuration), schema, Mode.CREATE, DEFAULT_BLOCK_SIZE, MAX_PADDING_SIZE_DEFAULT) ;
writer.start();
```
It creates to create parquet file in hadoop but can't facilitates to write List of Groups in parquet file
**ExampleParquetWriter**
```java
ParquetWriter writer = ExampleParquetWriter.builder(writePathFile).withConf(configuration).withType(getSchema()).build();
System.out.println("Number of groups to write:" + groups.size());
for (Group g : groups) {
writer.write(g);
}
```
Doesn't support for writing parquet file in hadoop
**Reporter**: [Udit Joshi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=udit0612)
#### Original Issue Attachments:
- [image-2019-05-31-10-41-45-674.png](https://issues.apache.org/jira/secure/attachment/12970426/image-2019-05-31-10-41-45-674.png)
**Note**: *This issue was originally created as [PARQUET-1587](https://issues.apache.org/jira/browse/PARQUET-1587). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing ParquetWriter, ParquetFileWriter, ExampleParquetWriter, and ParquetRecordWriter in the parquet-hadoop 1.10.1 API, along with the shown ExampleParquetWriter usage. Determine how GroupSupport is expected to work with Hadoop output and whether the inaccessible constructor is part of the needed path. Done means a supported way to write a list of Groups to a Hadoop Parquet file is available and demonstrated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100