Unable to write or read metadata from a Table after attaching metadata to the table
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Hey JuliaData team,
Recently ran into an odd bug where when I had an Arrow Table, I could not write that table to an `arrow` file after I had attached metadata to that table via the command `setmetadata`. My dictionary was correctly typed as `Dict(String, String)` and I was able to see my metadata attached to the table.
However, when I tried writing this table to a file, my REPL locked up and hung forever.Even after pressing C-c multiple times, it did not quit or stop processing but rather said `Warning: Force throwing a sigint" and still did not do anything. I had to manually close my terminal as it locked up the terminal.
Any thoughts as to why this could be happening? It was a rather large file so maybe that could've done something? I tried to reproduce this behavior below with a toy example:
```julia
using Arrow
using DataFrames
df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"])
Arrow.write("test.arrow", df)
at = Arrow.Table("test.arrow")
my_meta = Dict("A" => "Cool numbers", "B" => "Cool letters")
Arrow.setmetadata!(at, my_meta)
```
Which shows this as expected:
```julia
julia> at
Arrow.Table with 4 rows, 2 columns, and schema:
:A Int64
:B String
with metadata given by a Dict{String, String} with 2 entries:
"B" => "Cool letters"
"A" => "Cool numbers"
```
But then continuing on
```julia
Arrow.write("meta.arrow", at)
mat = Arrow.Table("meta.arrow")
```
produces the following:
```julia
julia> mat = Arrow.Table("meta.arrow")
Arrow.Table with 4 rows, 2 columns, and schema:
:A Int64
:B String
julia> Arrow.getmetadata(mat)
```
Where no metadata is shown. So, two different issues in two different ways I tried to get metadata saved to my data. Am I doing something wrong here? Thanks all!
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the supplied Julia reproduction with Arrow.setmetadata!, Arrow.write, and Arrow.getmetadata, checking both the write hang and the missing metadata after reload. Trace those entry points through the Arrow table serialization path. Done means writing a metadata-bearing table completes and Arrow.Table plus Arrow.getmetadata returns the attached values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100