apache / apache/arrow-julia

Serializing `Vector{Array}` flattens elements

Open
#125 5 comments 5 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

```
julia> tab2 = (x = [rand(2,2) for _ in 1:10], );

julia> io2 = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> Arrow.write(io2, tab2)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=736, maxsize=Inf, ptr=737, mark=-1)

julia> seekstart(io2)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=736, maxsize=Inf, ptr=1, mark=-1)

julia> tab2_in = Arrow.Table(io2)
Arrow.Table: (x = [[0.0854794818573692, 0.6156692467610922, 0.4816953048461168, 0.10732732583276494], [0.2699150559983352, 0.7190134247010207, 0.4620847068434688, 0.8086969500721466], [0.32599895190448236, 0.5913226835708243, 0.914412039189433, 0.04825070891100114], [0.19800356830304522, 0.30969840980302754, 0.7586621051179192, 0.8383737966971678], [0.46353328096692525, 0.7946259468169095, 0.9205855777866754, 0.30143398878132244], [0.7980760486814706, 0.3204359773634282, 0.5253557751574975, 0.8140218988518082], [0.9662048857122691, 0.17369565835410516, 0.18328845062159216, 0.5595773503180002], [0.5423562251958653, 0.6285041382249899, 0.31820822802763304, 0.42185178416765723], [0.6985239393699472, 0.8842247274247905, 0.5466899110422125, 0.2058087506887174], [0.352616487112392, 0.6636744200136235, 0.9140196029380305, 0.2594771305442407]],)

julia> tab2_in.x
10-element Arrow.List{Vector{Float64}, Int32, Arrow.Primitive{Float64, Vector{Float64}}}:
[0.0854794818573692, 0.6156692467610922, 0.4816953048461168, 0.10732732583276494]
[0.2699150559983352, 0.7190134247010207, 0.4620847068434688, 0.8086969500721466]
[0.32599895190448236, 0.5913226835708243, 0.914412039189433, 0.04825070891100114]
[0.19800356830304522, 0.30969840980302754, 0.7586621051179192, 0.8383737966971678]
[0.46353328096692525, 0.7946259468169095, 0.9205855777866754, 0.30143398878132244]
[0.7980760486814706, 0.3204359773634282, 0.5253557751574975, 0.8140218988518082]
[0.9662048857122691, 0.17369565835410516, 0.18328845062159216, 0.5595773503180002]
[0.5423562251958653, 0.6285041382249899, 0.31820822802763304, 0.42185178416765723]
[0.6985239393699472, 0.8842247274247905, 0.5466899110422125, 0.2058087506887174]
[0.352616487112392, 0.6636744200136235, 0.9140196029380305, 0.2594771305442407]
```

At first I thought this was just some cleverness with how Arrow was serializing a "column matrix" (e.g. a n-by-1 matrix). I guess it's probably not possible to store the metadata about the size of the elements if each element can potentially have its own size? Maybe using StaticArrays is called for?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Julia example with Arrow.write and Arrow.Table for a Vector{Array}. Investigate how the round trip represents each matrix element, and consider the issue complete when nested arrays retain their individual dimensions instead of being flattened.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.