apache / apache/parquet-java

Parquet Writer doesn't support to write list of Groups directly into hadoop

未关闭
#2,331 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Blocker Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

**Problem :-**  Parquet Writer doesn't support to write list of Groups directly into hadoop.

Cause of Problem :- Suitable classes are not available
Constructors are not visible

**All possible solution which i tried -**
- ParquetWriter
- ParquetFileWriter
- ExampleParquetWriter
- ParquetRecordWriter

**Gradle dependency**
> compile group: 'org.apache.parquet', name: 'parquet-hadoop', version: '1.10.1'
 

**Is there any way to writer parquet file into Hadoop with GroupSupport?**

 

**Problem description**

I am trying to create the object of ParquetWriter class which accepts the argument (OutputFile, Mode, WriteSupport, CompressionCodecName, int, boolean, Configuration, int, ParquetProperties). But this constructor has default access modifier. I can't able to access it.

 I also used ParquetFileWriter class but it doesn't show any group support.

 

**ParquetWriter**

**![image-2019-05-31-10-41-45-674.png](https://issues.apache.org/jira/secure/attachment/12970426/image-2019-05-31-10-41-45-674.png)**

 **ParquetFileWriter**
```java

ParquetFileWriter writer = new ParquetFileWriter(HadoopOutputFile.fromPath(writePathFile, configuration), schema, Mode.CREATE, DEFAULT_BLOCK_SIZE, MAX_PADDING_SIZE_DEFAULT) ;
writer.start();
```
 

 It creates  to create parquet file in hadoop but can't facilitates to write  List of Groups in parquet file

 

**ExampleParquetWriter**
```java

ParquetWriter writer = ExampleParquetWriter.builder(writePathFile).withConf(configuration).withType(getSchema()).build();
System.out.println("Number of groups to write:" + groups.size());
for (Group g : groups) {
writer.write(g);

```
 Doesn't support for writing  parquet file in hadoop

 

 

 

 

**Reporter**: [Udit Joshi](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=udit0612)
#### Original Issue Attachments:
- [image-2019-05-31-10-41-45-674.png](https://issues.apache.org/jira/secure/attachment/12970426/image-2019-05-31-10-41-45-674.png)

**Note**: *This issue was originally created as [PARQUET-1587](https://issues.apache.org/jira/browse/PARQUET-1587). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先查看 parquet-hadoop 1.10.1 API 中的 ParquetWriter、ParquetFileWriter、ExampleParquetWriter 和 ParquetRecordWriter,以及所示的 ExampleParquetWriter 用法。确定 GroupSupport 应如何与 Hadoop 输出配合工作,以及无法访问的构造函数是否属于所需路径的一部分。完成的标准是:提供并演示一种受支持的方式,将 Groups 列表写入 Hadoop Parquet 文件。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。