apache / apache/arrow-julia

Add an indexable variant of Arrow.Stream

Open
#353 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

In distributed computing context it would be nice to have a vector-variant of `Arrow.Stream` iterator. The idea is to be able to split processing of a single large arrow file with multiple record batches into multiple worker processes. Looking at the source code this should be possible to be done in a relatively efficient way.

@quinnj - what do you think?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing Arrow.Stream implementation and its source code to understand how record batches are exposed. Define the indexable vector variant and its worker-splitting behavior before implementation; done means a large Arrow file can be partitioned across multiple worker processes with appropriate coverage for the new API.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.