NTuple with custom type and compression
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I have a custom type defined this way:
```julia
struct Char8 <: AbstractChar
x::UInt8
end
Char8(x::Integer) = Char8(UInt8(x))
Base.codepoint(c::Char8) = UInt32(c.x)
```
and serialized this way
```julia
ArrowTypes.ArrowKind(::Type{Char8}) = ArrowTypes.PrimitiveKind()
ArrowTypes.ArrowType(::Type{Char8}) = UInt8
const CHAR8 = Symbol("JuliaLang.Char8")
ArrowTypes.arrowname(::Type{Char8}) = CHAR8
ArrowTypes.toarrow(x::Char8) = x.x
ArrowTypes.fromarrow(::Type{Char8}, x::UInt8) = Char8(x)
ArrowTypes.JuliaType(::Val{CHAR8}) = Char8
```
The following throws:
```julia
a=[(Char8(1),Char8(2))]
table = (col1=a,)
io = IOBuffer()
Arrow.write(io, table;compress=:zstd)
```
but only when the compression is enabled. Is that expected?
I also noticed that the ArrowType seems wrong, because it calls the identity function.
So, why not setting the following default,
```ArrowTypes.ArrowType(::Type{NTuple{N, T}}) where {N, T} = NTuple{N, ArrowTypes.ArrowType(T)}``` ?
This line solves this issue in my case.
Thanks,
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the shown NTuple table with a custom Julia Char8 type through Arrow.write using compress=:zstd, then trace ArrowTypes.ArrowType and the compression path. Compare the compressed and uncompressed behavior and determine whether the proposed NTuple default preserves the expected Arrow serialization; done means the reproduction no longer throws and the type mapping is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100