[pyarrow] compute.count_distinct segfaults or gives incorrect results when given a chunked string array greater than 2G in size
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
(2G IIRC, maybe 4G)
We have an automated process that gave incorrect results when trying to get the unique count of a large array. Debugging alternately, gave an incorrect count, or seg-faulted. The work around is to cast to a large-string array, but silently giving bad results or seg-faulting is obviously bad. :)
In general the low level (C++?) code should check for string-offset overflow. (I'm assuming a string array is being used under the hood for `count.distinct`.)
The Rust arrow code catches string-offset overflows in my experience, FWIW.
### Component(s)
C++, Python
Contributor guide
Assessment
This issue has not been assessed yet.