apache / apache/arrow-julia

Unable to write or read metadata from a Table after attaching metadata to the table

Open
#236 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Hey JuliaData team,

Recently ran into an odd bug where when I had an Arrow Table, I could not write that table to an `arrow` file after I had attached metadata to that table via the command `setmetadata`. My dictionary was correctly typed as `Dict(String, String)` and I was able to see my metadata attached to the table.

However, when I tried writing this table to a file, my REPL locked up and hung forever.Even after pressing C-c multiple times, it did not quit or stop processing but rather said `Warning: Force throwing a sigint" and still did not do anything. I had to manually close my terminal as it locked up the terminal.

Any thoughts as to why this could be happening? It was a rather large file so maybe that could've done something? I tried to reproduce this behavior below with a toy example:

```julia
using Arrow
using DataFrames

df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"])
Arrow.write("test.arrow", df)

at = Arrow.Table("test.arrow")

my_meta = Dict("A" => "Cool numbers", "B" => "Cool letters")

Arrow.setmetadata!(at, my_meta)
```

Which shows this as expected:

```julia
julia> at
Arrow.Table with 4 rows, 2 columns, and schema:
:A Int64
:B String

with metadata given by a Dict{String, String} with 2 entries:
"B" => "Cool letters"
"A" => "Cool numbers"
```

But then continuing on

```julia
Arrow.write("meta.arrow", at)

mat = Arrow.Table("meta.arrow")
```

produces the following:

```julia
julia> mat = Arrow.Table("meta.arrow")
Arrow.Table with 4 rows, 2 columns, and schema:
:A Int64
:B String

julia> Arrow.getmetadata(mat)

```

Where no metadata is shown. So, two different issues in two different ways I tried to get metadata saved to my data. Am I doing something wrong here? Thanks all!

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the supplied Julia reproduction with Arrow.setmetadata!, Arrow.write, and Arrow.getmetadata, checking both the write hang and the missing metadata after reload. Trace those entry points through the Arrow table serialization path. Done means writing a metadata-bearing table completes and Arrow.Table plus Arrow.getmetadata returns the attached values.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.