Reading Avro files supports other types
- 主要语言
- Rust
- 星标
- 9.3k
- 派生
- 2.4k
- 平均合并
- 3 天 11 小时
- 30 天内合并 PR
- 362
描述
### Is your feature request related to a problem or challenge?
I am now integrating [incubator-paimon](https://github.com/apache/incubator-paimon)(it is a streaming data lake platform), when reading the avro file, an exception message will appear: **expected avro schema to be a record**, because `AvroArrowArrayReader` only supports `AvroSchema::Record`, but the avro file format of paimon is Union type([code here](https://github.com/apache/incubator-paimon/blob/d2ddaa9bfb865a93b12c7a0294cf936071bbc08b/paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java#L224))
### Describe the solution you'd like
It would be better if the parsing format could be implemented by the user. The default implementation is still the current way, no problem.
### Describe alternatives you've considered
My current solution: [here](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L84) and [this](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L110)
### Additional context
_No response_
贡献指南
调研方向
从 AvroArrowArrayReader 和链接的 arrow_array_reader.rs 示例开始,了解当前的 Record-only 路径以及提议的由用户提供解析逻辑的方法。定义一个扩展点,在保留当前默认行为的同时,允许处理 Paimon 等来源的 Union Avro 输入,然后验证现有 reader 的行为和新路径。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- rust
- 领域
- data-engineering
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 25/100