apache / apache/arrow

[pyarrow] compute.count_distinct segfaults or gives incorrect results when given a chunked string array greater than 2G in size

Open
#42,200 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

(2G IIRC, maybe 4G)

We have an automated process that gave incorrect results when trying to get the unique count of a large array. Debugging alternately, gave an incorrect count, or seg-faulted. The work around is to cast to a large-string array, but silently giving bad results or seg-faulting is obviously bad. :)

In general the low level (C++?) code should check for string-offset overflow. (I'm assuming a string array is being used under the hood for `count.distinct`.)

The Rust arrow code catches string-offset overflows in my experience, FWIW.

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.