apache / apache/arrow-julia

Inconsistent handling of eltype Decimals.Decimal (with silent errors?)

Open
#335 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

First of all, thank you for the amazing package! I have noticed unexpected behaviour that I wanted to point out.

**Expected behaviour:** rational numbers like 1.0 and 0.1 will be represented as Float; they can be saved and loaded again.

**Actual behaviour:**
When writing column with eltype Decimals.Decimal, `Arrow.write(filename,df)` will give a method error (see below) and `Arrow.write(filename,df;compress=:lz4)` will complete without an error, but the resulting table is wrong when re-read (see MWE below).

I've had a quick look at the code base and I cannot see any type checks - are those left to the user / MethodErrors?

MWE:
```
using Decimals
using DataFrames, Arrow

df=DataFrame(:a=>[Decimal(2.0)])

# this will fail with error that Decimal cannot be saved
Arrow.write("test.feather", df)
# nested task error: MethodError: no method matching write(::IOBuffer, ::Decimals.Decimal)

# this will succeed
Arrow.write("test.feather", df;compress=:lz4)

# but the loaded dataframe will be rubbish
df2=Arrow.Table("test.feather")|>DataFrame
# 1×1 DataFrame
# Row │ a
# │ Float64
# ─────┼─────────────
# 1 │ 2.1509e-314

```

Error stack trace from Arrow.write() without a keyword argument:
> ERROR: TaskFailedException
Stacktrace:
[1] wait
@ ./task.jl:345 [inlined]
[2] close(writer::Arrow.Writer{IOStream})
@ Arrow ~/.julia/packages/Arrow/ZlMFU/src/write.jl:230
[3] open(::Arrow.var"#120#121"{DataFrame}, ::Type, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:file,), Tuple{Bool}}})
@ Base ./io.jl:386
[4] #write#119
@ ~/.julia/packages/Arrow/ZlMFU/src/write.jl:57 [inlined]
[5] write(file_path::String, tbl::DataFrame)
@ Arrow ~/.julia/packages/Arrow/ZlMFU/src/write.jl:56
[6] top-level scope
@ REPL[14]:1
> nested task error: MethodError: no method matching write(::IOBuffer, ::Decimals.Decimal)
Closest candidates are:
write(::IO, ::Any) at io.jl:672
write(::IO, ::Any, ::Any...) at io.jl:673
write(::Base.GenericIOBuffer, ::UInt8) at iobuffer.jl:442
...
Stacktrace:
[1] write(io::IOBuffer, x::Decimals.Decimal)
@ Base ./io.jl:672
[2] writearray(io::IOStream, #unused#::Type{Decimals.Decimal}, col::Vector{Union{Missing, Decimals.Decimal}})
@ Arrow ~/.julia/packages/Arrow/ZlMFU/src/utils.jl:50
[3] writebuffer(io::IOStream, col::Arrow.Primitive{Union{Missing, Decimals.Decimal}, Vector{Union{Missing, Decimals.Decimal}}}, alignment::Int64)
@ Arrow ~/.julia/packages/Arrow/ZlMFU/src/arraytypes/primitive.jl:102
[4] write(io::IOStream, msg::Arrow.Message, blocks::Tuple{Vector{Arrow.Block}, Vector{Arrow.Block}}, sch::Base.RefValue{Tables.Schema}, alignment::Int64)
@ Arrow ~/.julia/packages/Arrow/ZlMFU/src/write.jl:365
[5] macro expansion
@ ~/.julia/packages/Arrow/ZlMFU/src/write.jl:149 [inlined]
[6] (::Arrow.var"#122#124"{IOStream, Int64, Tuple{Vector{Arrow.Block}, Vector{Arrow.Block}}, Base.RefValue{Tables.Schema}, Arrow.OrderedChannel{Arrow.Message}})()
@ Arrow ./threadingconstructs.jl:258

**Package version**
[69666777] Arrow v2.3.0
[a93c6f00] DataFrames v1.3.4
[194296ae] LibPQ v1.14.0

**versioninfo()** (but it was the same on 1.7)
Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
OS: macOS (arm64-apple-darwin21.3.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
Threads: 6 on 6 virtual cores

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the MWE with Decimals.Decimal and compare Arrow.write behavior with and without compress=:lz4. Start in src/write.jl, src/utils.jl at writearray, and src/arraytypes/primitive.jl at writebuffer; inspect how unsupported values are handled. Done means the Decimal case behaves consistently and does not silently produce corrupted data.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.