Reading Avro files supports other types
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I am now integrating [incubator-paimon](https://github.com/apache/incubator-paimon)(it is a streaming data lake platform), when reading the avro file, an exception message will appear: **expected avro schema to be a record**, because `AvroArrowArrayReader` only supports `AvroSchema::Record`, but the avro file format of paimon is Union type([code here](https://github.com/apache/incubator-paimon/blob/d2ddaa9bfb865a93b12c7a0294cf936071bbc08b/paimon-format/src/main/java/org/apache/paimon/format/avro/AvroSchemaConverter.java#L224))
### Describe the solution you'd like
It would be better if the parsing format could be implemented by the user. The default implementation is still the current way, no problem.
### Describe alternatives you've considered
My current solution: [here](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L84) and [this](https://github.com/Asura7969/arrow-datafusion/blob/8e3b23561baded77c1604a936dea1f40d30ebd66/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L110)
### Additional context
_No response_
Contributor guide
Research direction
Start with AvroArrowArrayReader and the linked arrow_array_reader.rs examples to understand the current Record-only path and the proposed user-provided parsing approach. Define an extension point that preserves the current default while allowing Union Avro input such as Paimon’s, then verify both the existing reader behavior and the new path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100