Arrow changes data type from input in unexpected ways
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
From this MWE input is unrecognizable from output (the path to the Zarr file is public and can be run locally):
```
dc = Zarr.zopen("http://its-live-data.s3.amazonaws.com/datacubes/v02/N20E100/ITS_LIVE_vel_EPSG32647_G0120_X650000_Y3250000.zarr")
C = dc["satellite_img1"][:]
input = DataFrame([C,C],:auto)
Arrow.write("test.arrow", input)
output = Arrow.Table("test.arrow")
```
`input.x1` looks like this:
```
1460-element Vector{Zarr.MaxLengthStrings.MaxLengthString{2, UInt32}}:
"1A"
⋮
"8."
```
while `output.x1` looks like this:
```
1460-element Arrow.List{String, Int32, Vector{UInt8}}:
"1\0"
⋮
"\0\0"
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked MWE using Arrow.write and Arrow.Table, then compare the input.x1 and output.x1 types and values for the Zarr.MaxLengthString data. Determine the intended round-trip representation and verify that the serialized output preserves the input values and expected type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100