avro_to_arrow: Support in memory apache_avro Value's
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I have Avro values in memory parsed using the apache_avro crate. I would like to convert these and get a RecordBatch.
### Describe the solution you'd like
Currently AvroArrowArrayReader is private and the avro_to_arrow::Reader only accepts a io reader and calls the apache_avro reader internally. I would have to currently serialize my in memory values and pass the bytes to avro_to_arrow::Reader.
### Describe alternatives you've considered
_No response_
### Additional context
I have a fork with a hacky fix, I can make a draft PR if but the API was a bit weird with the lifetime the apache_avro::Reader required
Contributor guide
Research direction
Start by reading the private AvroArrowArrayReader and the public avro_to_arrow::Reader, focusing on how the apache_avro::Reader lifetime is currently handled. Define an API that accepts in-memory apache_avro values and produces a RecordBatch without serializing them; done means callers can use parsed values directly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100