apache / apache/datafusion

Improve Avro Reader Types Support

Open
#18,810 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Support of the various byte types is rudimentary and only exists for the plain case.
Binary types cannot be read from Fixed types
both Binary and LargeBinary are creating BinaryArray, likewise for Utf8 and LargeUtf8, in case it doesn't crash, that will force a cast when the ArrayData needs to be converted into to correct type later.

On the performance side, all binary/utf8 types there do a full copy of the data to an owned form, for binary-intensive files this is an absolute performance killer, and can be rather easily resolved by using Cow types

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the Avro reader's handling of binary, fixed, Utf8, and LargeUtf8 types, then inspect any existing reader tests. Done means fixed binary values and the distinct binary and UTF-8 types are read correctly without unnecessary full copies.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.