[Java] Implement converter between Arrow record batches and Avro records
- 主要语言
- Java
- 星标
- 94
- 派生
- 152
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 11
描述
It would be useful for applications which need convert Avro data to Arrow data.
This is an adapter which convert data with existing API (like JDBC adapter) rather than a native reader (like orc).
We implement this function through Avro java project, receiving param like Decoder/Schema/DatumReader of Avro and return VectorSchemaRoot. For each data type we have a consumer class as below to get Avro data and write it into vector to avoid boxing/unboxing (e.g. GenericRecord#get returns Object)
```java
public class AvroIntConsumer implements Consumer {
private final IntWriter writer;
public AvroIntConsumer(IntVector vector)
{ this.writer = new IntWriterImpl(vector); }
@Override
public void consume(Decoder decoder) throws IOException
{ writer.writeInt(decoder.readInt()); writer.setPosition(writer.getPosition() + 1); }
```
We intended to support primitive and complex types (null value represented via unions type with null type), size limit and field selection could be optional for users.
**Reporter**: [Ji Liu](https://issues.apache.org/jira/browse/ARROW-5845) / @tianchen92
#### Subtasks:
- [X] [[Java] Create Avro adapter module and add dependencies](https://github.com/apache/arrow/issues/22265)
- [X] [[Java] Initial implement to convert Avro record with primitive types](https://github.com/apache/arrow/issues/22278)
- [X] [[Java] Avro adapter implement simple Record type ](https://github.com/apache/arrow/issues/22396)
- [X] [[Java] Avro adapter support convert nullable value](https://github.com/apache/arrow/issues/22439)
- [X] [[Java] Avro adapter implement unions type](https://github.com/apache/arrow/issues/22496)
- [X] [[Java] Avro adapter avoid potential resource leak.](https://github.com/apache/arrow/issues/22590)
- [X] [[Java] Add API to avro adapter to limit number of rows returned at a time.](https://github.com/apache/arrow/issues/22608)
- [X] [[Java] Avro adapter implement Array/Map/Fixed type](https://github.com/apache/arrow/issues/22649)
- [X] [[Java] Avro adapter implement Enum type and nested Record type](https://github.com/apache/arrow/issues/16862)
- [X] [[Java] Add benchmark and large fake data UT for avro adapter](https://github.com/apache/arrow/issues/22830)
- [X] [[Java] Add support for skipping decoding of columns/field in Avro converter](https://github.com/apache/arrow/issues/22949)
- [ ] [[Java] Experiment with performance difference of avoiding the use of Avro Decoder](https://github.com/apache/arrow-java/issues/281)
- [X] [[Java] Support logical type encodings from Avro](https://github.com/apache/arrow/issues/22950)
- [ ] [[Java] Avro - Experiment with "compiled" consumer delegates for performance.](https://github.com/apache/arrow-java/issues/280)
- [X] [[JAVA] Avro adapter benchmark only runs once in JMH](https://github.com/apache/arrow/issues/23064)
- [X] [[Java] Extract a common base class for avro converter consumers](https://github.com/apache/arrow/issues/23236)
- [X] [[Java] Avro converter should convert attributes and props to FieldType metadata](https://github.com/apache/arrow/issues/23755)
**Note**: *This issue was originally created as [ARROW-5845](https://issues.apache.org/jira/browse/ARROW-5845). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
贡献指南
调研方向
查看已完成的子任务以了解现有的 Avro 适配器,然后检查剩余的两个性能实验:避免使用 Avro Decoder,以及使用已编译的 consumer delegates。该 issue 没有为这些实验列出实现文件、测试或验收标准,因此请在开始前与维护者确认预期范围。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 15/100