apache / apache/datafusion

Add support for complex nested types in List Arrays and Struct Arrays in `avro_to_arrow`

Open
#11,342 1 comment 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We are currently working on a stream processing system built atop DataFusion and as such Avro is a major format for us given its ubiquity in the Kafka world. We tried using the the existing Avro Reader in data fusion, however found it lacking in some critical ways that make not terribly useful for us in its present state.

The reader currently does not support complex nested datatypes such as -

1. The List arrays [only support primitive types](https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs#L627)
2. Dictionary arrays only support Utf8 as its value types.

Lastly, the reader seems to rely on `decode_internal` method on the `apache-avro` crate and seems to implement some of the Avro decoding "by hand". We ended up rolling our reader to support and we're able to use `decode_from_avro` datum and entirely pass on the avro decoding responsibility to the avro package.

Would love to work with @tustvold who seems to contributed here the most to augment the existing limitations here.

### Describe the solution you'd like

Addition of support for parsing complex datatypes.

### Describe alternatives you've considered

Convert avro > json then rely on json_to_arrow conversion, but this leads to inevitable loss of type information.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs at the List-array handling around line 627. Compare the existing decode_internal path with the requested decode_from_avro datum approach, then trace how Struct and Dictionary arrays are built. Done means complex nested values are parsed without losing type information, including the cases described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.