apache / apache/arrow-julia

Arrow changes data type from input in unexpected ways

Open
#359 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

From this MWE input is unrecognizable from output (the path to the Zarr file is public and can be run locally):

```
dc = Zarr.zopen("http://its-live-data.s3.amazonaws.com/datacubes/v02/N20E100/ITS_LIVE_vel_EPSG32647_G0120_X650000_Y3250000.zarr")
C = dc["satellite_img1"][:]
input = DataFrame([C,C],:auto)
Arrow.write("test.arrow", input)
output = Arrow.Table("test.arrow")
```

`input.x1` looks like this:
```
1460-element Vector{Zarr.MaxLengthStrings.MaxLengthString{2, UInt32}}:
"1A"

"8."
```

while `output.x1` looks like this:
```
1460-element Arrow.List{String, Int32, Vector{UInt8}}:
"1\0"

"\0\0"
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked MWE using Arrow.write and Arrow.Table, then compare the input.x1 and output.x1 types and values for the Zarr.MaxLengthString data. Determine the intended round-trip representation and verify that the serialized output preserves the input values and expected type.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.