apache / apache/arrow-julia

Set/retrieve ordered flag for DictEncoded

Open
#259 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

The Arrow dictionary format allows for an `ordered` flag indicating that the order of the elements in the dictionary should be preserved. This corresponds to the `ordered` property of a `CategoricalArray` in Julia. It would help to preserve the ordered property when writing the `DictEncoded` value and also to preserve it when reading. A conversion method to `CategoricalArray` could then set the ordered flag.

As an example of the current behavior, in Julia if I write an ordered CategoricalArray
```julia
julia> using Arrow, CategoricalArrays

julia> levs = ["d", "c", "b", "a"];

julia> a = CategoricalArray(repeat(levs, inner=3); levels = levs, ordered=true);

julia> b = CategoricalArray(repeat(levs, inner=3); levels = levs, ordered=false);

julia> Arrow.write("/tmp/test.arrow", (; a = a, b = b))
"/tmp/test.arrow"
```
both end up in python as unordered.
```python
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.feather as fthr
>>> fthr.read_table('/tmp/test.arrow')
pyarrow.Table
a: dictionary not null
b: dictionary not null
>>>
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how DictEncoded values are written and read, then inspect the conversion to CategoricalArray. Use the Julia example and pyarrow output in the issue to verify that ordered=true and ordered=false survive both serialization and conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.