apache / apache/arrow

Compute method if_else seems to create an invalid Dictionary Array

Open
#47,886 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When if_else() is performed on two ChunkedArrays of ordered dictionary type, the resultant ChunkedArray is somewhat strange:
```python
import pyarrow as pa
import pyarrow.compute as pc
dictionary = ["a", "b", "c"]

c1 = pa.DictionaryArray.from_arrays(pa.array([0,1,0], type=pa.int32()), dictionary, ordered=True)
c2 = pa.DictionaryArray.from_arrays(pa.array([2,1,0], type=pa.int32()), dictionary, ordered=True)
c3 = pa.DictionaryArray.from_arrays(pa.array([1,1,0], type=pa.int32()), dictionary, ordered=True)
left = pa.chunked_array([c1, c2, c3])
print(left.type) # ordered=True

c1 = pa.DictionaryArray.from_arrays(pa.array([0,1,0], type=pa.int32()), dictionary, ordered=True)
c2 = pa.DictionaryArray.from_arrays(pa.array([2,None,0], type=pa.int32()), dictionary, ordered=True)
c3 = pa.DictionaryArray.from_arrays(pa.array([None,1,0], type=pa.int32()), dictionary, ordered=True)
right = pa.chunked_array([c1, c2, c3])
print(right.type) # ordered=True

mask = pc.is_null(right)
res = pc.if_else(mask, left, right)
print(res.type) # ordered=True
print(res.chunk(0).type) # ordered=False
print(res.chunk(1).type) # ordered=False
print(res.chunk(2).type) # ordered=False
```

`res.type` shows ordered=True, while the internal chunks are unordered.
If internal chunks are `unordered`, should not the ChunkedArray (res) be marked as `unordered` as well?

It causes some internal issues in my application (I am using C++ interface: arrow::compute::CallFunction) aborting my program with: `Array chunks must all be same type`.

PS. I checked with nightly package and it seems to be there as well:
`pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pyarrow`

Looking forward to your input/suggestion on this.

Thanks and Regards,
Sourav Saha

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.