apache / apache/arrow

[C++][Python] sort_by produces incorrect result

Open
#45,686 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Critical Fix Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

To reproduce:

```python
import pyarrow
import numpy as np

print(pyarrow.__version__)

N = 2**30 // 4

data = {
"id": [4, 3, 2, 1],
"data": [np.zeros(N, dtype=np.int64) + i for i in range(4)],
}
table = pyarrow.Table.from_pydict(data)

table2 = table.sort_by("id")
print(table2)
```

Actual output:

```
pyarrow.__version__ = '19.0.1'
pyarrow.Table
id: int64
data: list
child 0, item: int64
----
id: [[1,2,3,4]]
data: [[[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0],[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0]]]
```

Changing dtype to `np.int8` gives the correct output:

```
pyarrow.__version__ = '19.0.1'
pyarrow.Table
id: int64
data: list
child 0, item: int8
----
id: [[1,2,3,4]]
data: [[[3,3,3,3,3,...,3,3,3,3,3],[2,2,2,2,2,...,2,2,2,2,2],[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0]]]
```

My guess is that it overflows when calculating offsets during sorting, although I have no idea how pyarrow works internally.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.