apache / apache/arrow

[pyarrow] `pyarrow.compute.unique` gives garbage results with chunked dictionary arrays.

Open
#45,010 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Consider:
```
In [9]: pyarrow.__version__
Out[9]: '18.0.0'

In [10]: a = pa.DictionaryArray.from_arrays(pa.array([63], type='int8'), [f"a{i}" for i in range(64)])

In [11]: b = pa.DictionaryArray.from_arrays(pa.array([64], type='int8'), [f"b{i}" for i in range(65)])

In [12]: c = pa.chunked_array([a, b])

In [13]: pa.compute.unique(c).indices
Out[13]:

[
63,
-128 # <- bad results.
]

In [14]:
```

We have a chunked array where each chunk is a dictionary array. For such cases, seems like `unique` is trying to fit the final result into input type which is leading to wraparound / garbage values in the result.

Even calling `validate` on above result does not error out.
```
In [10]: pa.compute.unique(c).validate()

In [11]:
```

Should `unique` just raise an error for such cases or should it try to fit the output into a suitable type?

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.