apache / apache/datafusion

Reading Avro files supports other types

未关闭
#7,828 5 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Rust
星标
9.3k
派生
2.4k
平均合并
3 天 11 小时
30 天内合并 PR
362

描述

### Is your feature request related to a problem or challenge?

I am now integrating [incubator-paimon](https://github.com/apache/incubator-paimon)(it is a streaming data lake platform), when reading the avro file, an exception message will appear: **expected avro schema to be a record**, because `AvroArrowArrayReader` only supports `AvroSchema::Record`, but the avro file format of paimon is Union type([code here](https://github.com/apache/incubator-paimon/blob/d2ddaa9bfb865a93b12c7a0294cf936071bbc08b/paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java#L224))

### Describe the solution you'd like

It would be better if the parsing format could be implemented by the user. The default implementation is still the current way, no problem.

### Describe alternatives you've considered

My current solution: [here](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L84) and [this](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L110)

### Additional context

_No response_

贡献指南

打开贡献指南

调研方向

从 AvroArrowArrayReader 和链接的 arrow_array_reader.rs 示例开始,了解当前的 Record-only 路径以及提议的由用户提供解析逻辑的方法。定义一个扩展点,在保留当前默认行为的同时,允许处理 Paimon 等来源的 Union Avro 输入,然后验证现有 reader 的行为和新路径。

由索引模型根据 Issue 内容生成。

评估

技术栈
rust
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。