apache / apache/arrow-julia

Question on `Date` encoding

Open
#303 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I'm trying to use Arrow to send data between a Julia (Arrows.jl) and a Rust (Polars) app.
However, when I write a table containing Date, it is read by Polars as Extension("JuliaLang.Date", Date32, Some("")), and Polars complains with
```
Cannot create polars series from Extension("JuliaLang.Date", Date32, Some("")) type
```
I would have expected one of the following things would happen:

1. When writing the Arrow file, the type is converted to a suitable Arrow type (e.g. Date32 ),disguising its origin.
2. When reading the Arrow file, Polars would ignore the origin (JuliaLang.Date, which seems to just be a name) and recognize it as Date32 or whatever.

Instead, what seems to happen is that it is encoded as an extension type and Polars does not know what to do with it. Is this expected behavior?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported case: write a table containing `Date` with Arrows.jl and try to read it in Polars; the issue reports an `Extension("JuliaLang.Date", Date32, Some(""))` type error. The issue does not identify files or tests, so first establish whether the intended behavior is to preserve or ignore the extension metadata. Done means the expected behavior and an actionable change are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.