Add an indexable variant of Arrow.Stream
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
In distributed computing context it would be nice to have a vector-variant of `Arrow.Stream` iterator. The idea is to be able to split processing of a single large arrow file with multiple record batches into multiple worker processes. Looking at the source code this should be possible to be done in a relatively efficient way.
@quinnj - what do you think?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing Arrow.Stream implementation and its source code to understand how record batches are exposed. Define the indexable vector variant and its worker-splitting behavior before implementation; done means a large Arrow file can be partitioned across multiple worker processes with appropriate coverage for the new API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100