apache / apache/arrow-julia

NTuple with custom type and compression

Open
#281 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Hello,

I have a custom type defined this way:

```julia
struct Char8 <: AbstractChar
x::UInt8
end
Char8(x::Integer) = Char8(UInt8(x))
Base.codepoint(c::Char8) = UInt32(c.x)
```
and serialized this way
```julia
ArrowTypes.ArrowKind(::Type{Char8}) = ArrowTypes.PrimitiveKind()
ArrowTypes.ArrowType(::Type{Char8}) = UInt8
const CHAR8 = Symbol("JuliaLang.Char8")
ArrowTypes.arrowname(::Type{Char8}) = CHAR8
ArrowTypes.toarrow(x::Char8) = x.x
ArrowTypes.fromarrow(::Type{Char8}, x::UInt8) = Char8(x)
ArrowTypes.JuliaType(::Val{CHAR8}) = Char8
```

The following throws:
```julia
a=[(Char8(1),Char8(2))]
table = (col1=a,)
io = IOBuffer()
Arrow.write(io, table;compress=:zstd)
```
but only when the compression is enabled. Is that expected?

I also noticed that the ArrowType seems wrong, because it calls the identity function.
So, why not setting the following default,
```ArrowTypes.ArrowType(::Type{NTuple{N, T}}) where {N, T} = NTuple{N, ArrowTypes.ArrowType(T)}``` ?
This line solves this issue in my case.

Thanks,

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the shown NTuple table with a custom Julia Char8 type through Arrow.write using compress=:zstd, then trace ArrowTypes.ArrowType and the compression path. Compare the compressed and uncompressed behavior and determine whether the proposed NTuple default preserves the expected Arrow serialization; done means the reproduction no longer throws and the type mapping is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.