apache / apache/arrow-julia

colmetadata does not read custom metadata with multiple writes

Open
#485 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

This problem is described in more detail in [discourse](https://discourse.julialang.org/t/reading-and-writing-arrow-column-metadata/102427).

Specifically the following MWE works when the for loop executes once, but the colmetadata gives nothing when the for loop executes more than once.

`using Arrow
using Tables
# go to runtest.jl in Arrow package for hints on how to use metadata and colmetadata

cd(@__DIR__)
filename = "test10.arrow"
meta = ["file" => "test.arrow", "when" => "now", "why" => "gain experience"]
metacol1 = ["id" => "chan1", "comment" => "Chan 1 comment"]
metacol2 = ["id" => "chan2", "comment" => "Chan 2 comment"]
writer = open(Arrow.Writer, filename, metadata=meta, colmetadata = Dict(:Column1 => metacol1, :Column2 => metacol2))

for i in 1:1
result = rand(2, 2) .+ i
Arrow.write(writer, Tables.table(result)) #, header=[:a,:b])
end
close(writer)

tbl = Arrow.Table(filename)

tblmeta = Arrow.getmetadata(tbl)
tblmeta
tblmeta["why"]
tblmeta["when"]

tblmetachan1 = Arrow.getmetadata(tbl.Column1)
tblmetachan1
tblmetachan1 == nothing
`

For once through the for loop we get
`julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)
Base.ImmutableDict{String, String} with 2 entries:
"comment" => "Chan 1 comment"
"id" => "chan1"
`

When we go through the for loop twice (or more) we get
`julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)

julia> tblmetachan1

julia> tblmetachan1 == nothing
true`

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the multi-write MWE using Arrow.Writer, Arrow.write, Arrow.Table, and Arrow.getmetadata; the issue points to runtest.jl for usage hints. Trace how column metadata is written and read across repeated writes. Done means tbl.Column1 and tbl.Column2 retain their metadata after the loop runs more than once.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.