[JAVA][C++]Support Parquet Read and Write in Java
- 主要语言
- Java
- 星标
- 94
- 派生
- 152
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 11
描述
We added a new java interface to support parquet read and write from hdfs or local file.
The purpose of this implementation is that when we loading and dumping parquet data in Java, we can only use rowBased put and get methods. Since arrow already has C++ implementation to load and dump parquet, so we wrapped those codes as Java APIs.
After test, we noticed in our workload, performance improved more than 2x comparing with rowBased load and dump. So we want to contribute codes to arrow.
since this is a total independent change, there is no codes change to current arrow codes. We added two folders as listed: java/adapter/parquet and cpp/src/jni/parquet
**Reporter**: [Chendi.Xue](https://issues.apache.org/jira/browse/ARROW-6720)
#### Related issues:
- [[Java][Dataset] Implement Datasets Java API ](https://github.com/apache/arrow/issues/17055) (incorporates)
- [[Java][Dataset] Support writing to files within dataset scanner via JNI](https://github.com/apache/arrow/issues/27628) (incorporates)
#### PRs and other links:
- [GitHub Pull Request apache/arrow#5522](https://github.com/apache/arrow/pull/5522)
- [GitHub Pull Request apache/arrow#5717](https://github.com/apache/arrow/pull/5717)
- [GitHub Pull Request apache/arrow#5719](https://github.com/apache/arrow/pull/5719)
**Note**: *This issue was originally created as [ARROW-6720](https://issues.apache.org/jira/browse/ARROW-6720). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
贡献指南
调研方向
首先检查 java/adapter/parquet 和 cpp/src/jni/parquet,然后将链接的 pull request #5522、#5717 和 #5719 与相关的 dataset issue 进行比较。检查用于在 HDFS 和本地文件中读写 Parquet 的 Java API 及其测试;当所需支持已存在且没有重复已纳入的 dataset 工作时,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- cpp, java
- 领域
- backend, data
- Issue 类型
- 功能
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 20/100