apache / apache/arrow-java

[Java] Implement converter between Arrow record batches and Avro records

オープン
#404 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る
Type: enhancement
主要言語
Java
スター
94
フォーク
152
平均マージ
3日 16時間
マージ済み PR(30日)
11

説明

It would be useful for applications which need convert Avro data to Arrow data.

This is an adapter which convert data with existing API (like JDBC adapter) rather than a native reader (like orc).

We implement this function through Avro java project, receiving param like Decoder/Schema/DatumReader of Avro and return VectorSchemaRoot. For each data type we have a consumer class as below to get Avro data and write it into vector to avoid boxing/unboxing (e.g. GenericRecord#get returns Object)
```java

public class AvroIntConsumer implements Consumer {
private final IntWriter writer;

public AvroIntConsumer(IntVector vector)

{ this.writer = new IntWriterImpl(vector); }

@Override
public void consume(Decoder decoder) throws IOException

{ writer.writeInt(decoder.readInt()); writer.setPosition(writer.getPosition() + 1); }
```
We intended to support primitive and complex types (null value represented via unions type with null type), size limit and field selection could be optional for users. 

**Reporter**: [Ji Liu](https://issues.apache.org/jira/browse/ARROW-5845) / @tianchen92
#### Subtasks:
- [X] [[Java] Create Avro adapter module and add dependencies](https://github.com/apache/arrow/issues/22265)
- [X] [[Java] Initial implement to convert Avro record with primitive types](https://github.com/apache/arrow/issues/22278)
- [X] [[Java] Avro adapter implement simple Record type ](https://github.com/apache/arrow/issues/22396)
- [X] [[Java] Avro adapter support convert nullable value](https://github.com/apache/arrow/issues/22439)
- [X] [[Java] Avro adapter implement unions type](https://github.com/apache/arrow/issues/22496)
- [X] [[Java] Avro adapter avoid potential resource leak.](https://github.com/apache/arrow/issues/22590)
- [X] [[Java] Add API to avro adapter to limit number of rows returned at a time.](https://github.com/apache/arrow/issues/22608)
- [X] [[Java] Avro adapter implement Array/Map/Fixed type](https://github.com/apache/arrow/issues/22649)
- [X] [[Java] Avro adapter implement Enum type and nested Record type](https://github.com/apache/arrow/issues/16862)
- [X] [[Java] Add benchmark and large fake data UT for avro adapter](https://github.com/apache/arrow/issues/22830)
- [X] [[Java] Add support for skipping decoding of columns/field in Avro converter](https://github.com/apache/arrow/issues/22949)
- [ ] [[Java] Experiment with performance difference of avoiding the use of Avro Decoder](https://github.com/apache/arrow-java/issues/281)
- [X] [[Java] Support logical type encodings from Avro](https://github.com/apache/arrow/issues/22950)
- [ ] [[Java] Avro - Experiment with "compiled" consumer delegates for performance.](https://github.com/apache/arrow-java/issues/280)
- [X] [[JAVA] Avro adapter benchmark only runs once in JMH](https://github.com/apache/arrow/issues/23064)
- [X] [[Java] Extract a common base class for avro converter consumers](https://github.com/apache/arrow/issues/23236)
- [X] [[Java] Avro converter should convert attributes and props to FieldType metadata](https://github.com/apache/arrow/issues/23755)

**Note**: *This issue was originally created as [ARROW-5845](https://issues.apache.org/jira/browse/ARROW-5845). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

完了したサブタスクを確認して既存の Avro アダプターを理解し、その後、残っている 2 つのパフォーマンス実験を調査します。Avro Decoder を回避することと、コンパイル済みの consumer delegates を使用することです。この issue には、これらの実験に関する実装ファイル、テスト、受け入れ基準が記載されていないため、開始前に maintainers と想定されるスコープを確認してください。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
15/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。