apache / apache/arrow-java

[JAVA][C++]Support Parquet Read and Write in Java

未關閉
#279 8 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
Type: enhancement
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

We added a new java interface to support parquet read and write from hdfs or local file.

The purpose of this implementation is that when we loading and dumping parquet data in Java, we can only use rowBased put and get methods. Since arrow already has C++ implementation to load and dump parquet, so we wrapped those codes as Java APIs.

After test, we noticed in our workload, performance improved more than 2x comparing with rowBased load and dump. So we want to contribute codes to arrow.

since this is a total independent change, there is no codes change to current arrow codes. We added two folders as listed:  java/adapter/parquet and cpp/src/jni/parquet

**Reporter**: [Chendi.Xue](https://issues.apache.org/jira/browse/ARROW-6720)
#### Related issues:
- [[Java][Dataset] Implement Datasets Java API ](https://github.com/apache/arrow/issues/17055) (incorporates)
- [[Java][Dataset] Support writing to files within dataset scanner via JNI](https://github.com/apache/arrow/issues/27628) (incorporates)
#### PRs and other links:
- [GitHub Pull Request apache/arrow#5522](https://github.com/apache/arrow/pull/5522)
- [GitHub Pull Request apache/arrow#5717](https://github.com/apache/arrow/pull/5717)
- [GitHub Pull Request apache/arrow#5719](https://github.com/apache/arrow/pull/5719)

**Note**: *This issue was originally created as [ARROW-6720](https://issues.apache.org/jira/browse/ARROW-6720). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

貢獻指南

開啟貢獻指南

研究方向

首先檢查 java/adapter/parquet 和 cpp/src/jni/parquet,接著將連結的 pull request #5522、#5717 和 #5719 與相關的 dataset issue 進行比較。檢查用於在 HDFS 和本機檔案中讀寫 Parquet 的 Java API 及其測試;當所需支援已存在,且沒有重複已納入的 dataset 工作時,即視為完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
cpp, java
領域
backend, data
Issue 類型
功能
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
20/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。