Improve Avro Reader Types Support
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Support of the various byte types is rudimentary and only exists for the plain case.
Binary types cannot be read from Fixed types
both Binary and LargeBinary are creating BinaryArray, likewise for Utf8 and LargeUtf8, in case it doesn't crash, that will force a cast when the ArrayData needs to be converted into to correct type later.
On the performance side, all binary/utf8 types there do a full copy of the data to an owned form, for binary-intensive files this is an absolute performance killer, and can be rather easily resolved by using Cow types
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by locating the Avro reader's handling of binary, fixed, Utf8, and LargeUtf8 types, then inspect any existing reader tests. Done means fixed binary values and the distinct binary and UTF-8 types are read correctly without unnecessary full copies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100