colmetadata does not read custom metadata with multiple writes
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
This problem is described in more detail in [discourse](https://discourse.julialang.org/t/reading-and-writing-arrow-column-metadata/102427).
Specifically the following MWE works when the for loop executes once, but the colmetadata gives nothing when the for loop executes more than once.
`using Arrow
using Tables
# go to runtest.jl in Arrow package for hints on how to use metadata and colmetadata
cd(@__DIR__)
filename = "test10.arrow"
meta = ["file" => "test.arrow", "when" => "now", "why" => "gain experience"]
metacol1 = ["id" => "chan1", "comment" => "Chan 1 comment"]
metacol2 = ["id" => "chan2", "comment" => "Chan 2 comment"]
writer = open(Arrow.Writer, filename, metadata=meta, colmetadata = Dict(:Column1 => metacol1, :Column2 => metacol2))
for i in 1:1
result = rand(2, 2) .+ i
Arrow.write(writer, Tables.table(result)) #, header=[:a,:b])
end
close(writer)
tbl = Arrow.Table(filename)
tblmeta = Arrow.getmetadata(tbl)
tblmeta
tblmeta["why"]
tblmeta["when"]
tblmetachan1 = Arrow.getmetadata(tbl.Column1)
tblmetachan1
tblmetachan1 == nothing
`
For once through the for loop we get
`julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)
Base.ImmutableDict{String, String} with 2 entries:
"comment" => "Chan 1 comment"
"id" => "chan1"
`
When we go through the for loop twice (or more) we get
`julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)
julia> tblmetachan1
julia> tblmetachan1 == nothing
true`
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the multi-write MWE using Arrow.Writer, Arrow.write, Arrow.Table, and Arrow.getmetadata; the issue points to runtest.jl for usage hints. Trace how column metadata is written and read across repeated writes. Done means tbl.Column1 and tbl.Column2 retain their metadata after the loop runs more than once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100