Add support for complex nested types in List Arrays and Struct Arrays in `avro_to_arrow`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We are currently working on a stream processing system built atop DataFusion and as such Avro is a major format for us given its ubiquity in the Kafka world. We tried using the the existing Avro Reader in data fusion, however found it lacking in some critical ways that make not terribly useful for us in its present state.
The reader currently does not support complex nested datatypes such as -
1. The List arrays [only support primitive types](https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L627)
2. Dictionary arrays only support Utf8 as its value types.
Lastly, the reader seems to rely on `decode_internal` method on the `apache-avro` crate and seems to implement some of the Avro decoding "by hand". We ended up rolling our reader to support and we're able to use `decode_from_avro` datum and entirely pass on the avro decoding responsibility to the avro package.
Would love to work with @tustvold who seems to contributed here the most to augment the existing limitations here.
### Describe the solution you'd like
Addition of support for parsing complex datatypes.
### Describe alternatives you've considered
Convert avro > json then rely on json_to_arrow conversion, but this leads to inevitable loss of type information.
### Additional context
_No response_
Contributor guide
Research direction
Start in datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs at the List-array handling around line 627. Compare the existing decode_internal path with the requested decode_from_avro datum approach, then trace how Struct and Dictionary arrays are built. Done means complex nested values are parsed without losing type information, including the cases described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100