apache / apache/arrow

[C++] combine_chunks on DictionaryArray appears to be broken

Open
#33,112 4 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Recently upgraded from pyarrow 4.0.1 to 9.0.0 and there appears to be a bug when combining the chunks of a dictionary with multiple row groups.  The dictionary is a stringarray of categories.

It is worth noting here that each category is not present in each chunk. To me, the issue appears to be that the category indices per chunk appear to be incorrect when a category is missing from a chunk when they are combined together. I assume this as counts for the categories of a lower index (0, 1) appear to be more frequent in the bugged version compared to the working version, and the counts of the lower indices (2, 3, 4) are lower.

 

The difference can be easily noted when running a value count. For example;

![two.png](https://issues.apache.org/jira/secure/attachment/13049954/two.png)

A workaround for now is to read directly as a string array, and then encode this as a dictionary. This isn't the best however due to speed and memory concerns.

![one.png](https://issues.apache.org/jira/secure/attachment/13049955/one.png)

 

Attached is my parquet file (test.parquet) and a simply python script to see the difference (category_counts.py). I did not create this parquet file, rather am consuming it from a service- so excuse the data / uuid style column names. Please run this with pyarrow 4.0.1 and pyarrow 9.0.0 to see the difference in output. The images say pyarrow 6.0.0 but the issue is still present in 9.0.0. too

 

**Reporter**: [Jared Weston](https://issues.apache.org/jira/browse/ARROW-17900)
#### Original Issue Attachments:
- [category_counts.py](https://issues.apache.org/jira/secure/attachment/13049953/category_counts.py)
- [one.png](https://issues.apache.org/jira/secure/attachment/13049955/one.png)
- [test.parquet](https://issues.apache.org/jira/secure/attachment/13049952/test.parquet)
- [two.png](https://issues.apache.org/jira/secure/attachment/13049954/two.png)

**Note**: *This issue was originally created as [ARROW-17900](https://issues.apache.org/jira/browse/ARROW-17900). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the attached category_counts.py and test.parquet, running the reproduction against pyarrow 4.0.1 and 9.0.0 to compare value counts. Trace the C++ combine_chunks path for DictionaryArray when categories are absent from individual chunks; done means combined indices produce the same category counts as the working version without converting through a string array.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.