Serializing `Vector{Array}` flattens elements
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
```
julia> tab2 = (x = [rand(2,2) for _ in 1:10], );
julia> io2 = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> Arrow.write(io2, tab2)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=736, maxsize=Inf, ptr=737, mark=-1)
julia> seekstart(io2)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=736, maxsize=Inf, ptr=1, mark=-1)
julia> tab2_in = Arrow.Table(io2)
Arrow.Table: (x = [[0.0854794818573692, 0.6156692467610922, 0.4816953048461168, 0.10732732583276494], [0.2699150559983352, 0.7190134247010207, 0.4620847068434688, 0.8086969500721466], [0.32599895190448236, 0.5913226835708243, 0.914412039189433, 0.04825070891100114], [0.19800356830304522, 0.30969840980302754, 0.7586621051179192, 0.8383737966971678], [0.46353328096692525, 0.7946259468169095, 0.9205855777866754, 0.30143398878132244], [0.7980760486814706, 0.3204359773634282, 0.5253557751574975, 0.8140218988518082], [0.9662048857122691, 0.17369565835410516, 0.18328845062159216, 0.5595773503180002], [0.5423562251958653, 0.6285041382249899, 0.31820822802763304, 0.42185178416765723], [0.6985239393699472, 0.8842247274247905, 0.5466899110422125, 0.2058087506887174], [0.352616487112392, 0.6636744200136235, 0.9140196029380305, 0.2594771305442407]],)
julia> tab2_in.x
10-element Arrow.List{Vector{Float64}, Int32, Arrow.Primitive{Float64, Vector{Float64}}}:
[0.0854794818573692, 0.6156692467610922, 0.4816953048461168, 0.10732732583276494]
[0.2699150559983352, 0.7190134247010207, 0.4620847068434688, 0.8086969500721466]
[0.32599895190448236, 0.5913226835708243, 0.914412039189433, 0.04825070891100114]
[0.19800356830304522, 0.30969840980302754, 0.7586621051179192, 0.8383737966971678]
[0.46353328096692525, 0.7946259468169095, 0.9205855777866754, 0.30143398878132244]
[0.7980760486814706, 0.3204359773634282, 0.5253557751574975, 0.8140218988518082]
[0.9662048857122691, 0.17369565835410516, 0.18328845062159216, 0.5595773503180002]
[0.5423562251958653, 0.6285041382249899, 0.31820822802763304, 0.42185178416765723]
[0.6985239393699472, 0.8842247274247905, 0.5466899110422125, 0.2058087506887174]
[0.352616487112392, 0.6636744200136235, 0.9140196029380305, 0.2594771305442407]
```
At first I thought this was just some cleverness with how Arrow was serializing a "column matrix" (e.g. a n-by-1 matrix). I guess it's probably not possible to store the metadata about the size of the elements if each element can potentially have its own size? Maybe using StaticArrays is called for?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Julia example with Arrow.write and Arrow.Table for a Vector{Array}. Investigate how the round trip represents each matrix element, and consider the issue complete when nested arrays retain their individual dimensions instead of being flattened.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100