apache / apache/arrow-java

[Java] Port Row Set abstraction from Drill to Arrow

未關閉
#372 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
Type: enhancement
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

Arrow is a great way to exchange data between systems. Somewhere in the process, however, data must be load into, and read out of the Arrow vectors.

Arrow's vector code started with similar code in Apache Drill. The Drill project created a "Row Set" abstraction that:
- Provides a simple way to define the schema for a set of batches.
- Loads data into vectors from row-oriented inputs.
- Reads data out of vectors in row-oriented output.
- Controls memory consumed by the record batch when loading data into vectors.
- Ensures maximum usage of the allocated vector space when loading data Into vectors.
- Optionally handles projection when reading data from an input file into a set of vectors.
- Optionally handles data conversion from input to vector formats.

This mechanism is handy for any Java developer who produces or consumes Arrow vectors.

Detailed information is available in [this wiki](https://github.com/paul-rogers/arrow/wiki), including a more detailed description of the motivation for this project, and an analysis of the work required to do the Drill-to-Arrow port.

The code is in Java simply because Drill is written in Java. The same mechanisms can be ported to other languages if useful. Those ports would be separate future projects.

The code will be placed in a new Java module which can be imported by projects that wish to use the code. Changes may be needed to expose items from the `vector` module; we'll tackle those issues if/when they occur.

**Reporter**: [Paul Rogers](https://issues.apache.org/jira/browse/ARROW-3164)

**Note**: *This issue was originally created as [ARROW-3164](https://issues.apache.org/jira/browse/ARROW-3164). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

貢獻指南

開啟貢獻指南

研究方向

從連結的 Arrow wiki 開始,尤其關注其中的動機和 Drill-to-Arrow 移植分析。檢查提議的新 Java 模組應如何與現有的 vector 模組互動,並確認任何所需的公開範圍變更。當 Row Set 抽象被移植到一個可匯入的 Java 模組中,並具備所描述的 schema、載入、讀取、記憶體、投影和轉換能力時,即視為完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
data
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
需要釐清
新手友好度
20/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。