apache / apache/arrow-julia

Arrow stream writer and reader implementation questions

Open
#475 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I was using the python arrow function record_batch and find that this does not exist in julia, but there is an arrow table function. However they don’t behave the same way.

I have a customer requirement (that I can neither explain nor defend) of using arrow to
make a stream writer and reader of io bytes rather than just use julia bytes
This works using python arrows and batch_record
batch = pa.record_batch([data_in],names=[‘f0’])

My question is, is the problem syntax (typing it wrong) or semantics (just not possible to do because functionality was not ported)?

my julia code is as follows:
blksz = (32 * 1024)

A = rand(ComplexF16, blksz)

B = reinterpret(Float16, A)

C = reinterpret(UInt8, B)

tbl = Arrow.Table(C)

→ MethodError: no method matching open(::Base.ReinterpretArray{UInt8, 1, ComplexF16, Vector{ComplexF16}, false}, ::String

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing Julia's Arrow.Table entry point with the Python record_batch usage shown in the issue, then reproduce the MethodError with the provided reinterpretation example. Determine whether stream readers and writers for IO bytes are supported; done should clarify the supported API or define the missing functionality needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.