apache / apache/arrow

[Python] combine_chunks fails on column of table, but does not error on table itself

Open
#28,850 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

combine_chunks fails on column of table, but does not error on table itself (but creates 3 chunks instead).

Is there a reason why they are not handled the same?
```python

In [90]: pa.__version__
Out[90]: '4.0.0'

# Get shape
In [85]: pa_table.shape
Out[85]: (102753589, 1)In [86]: pa_col1_array = pa_table.column(0)

# Get number of chunks
In [87]: pa_col1_array.num_chunks
Out[87]: 4404

# Combining chunks on the pyarrow table with one column works.
In [88]: pa_table.combine_chunks()
Out[88]:
pyarrow.Table
# id=TEW__014e25__c14e1d__Multiome_RNA_brain_10x_no_perm: string

# Combining chunks on the column itself does not work.
In [89]: pa_col1_array.combine_chunks()
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
----> 1 pa_col1_array.combine_chunks()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.combine_chunks()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: offset overflow while concatenating arrays

# Assign combine chunks table to new tabled.
In [91]: pa_table_combined = pa_table.combine_chunks()

# Get first column
In [92]: pa_col1_array_from_pa_table_combined = pa_table_combined.column(0)

# Get number of chunks
In [93]: pa_col1_array_from_pa_table_combined.num_chunks
Out[93]: 3

# Try to combine column 1 again.
In [94]: pa_col1_array_from_pa_table_combined.combine_chunks()
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
----> 1 pa_col1_array_from_pa_table_combined.combine_chunks()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.combine_chunks()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
/software/miniconda3/envs/cisTopic/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: offset overflow while concatenating arrays

# Get sizes of each chunk.
In [106]: [chunk.nbytes for chunk in pa_col1_array_from_pa_table_combined.chunks]
Out[106]: [2341650593, 2342925682, 241257842]
```

**Reporter**: [Gert Hulselmans](https://issues.apache.org/jira/browse/ARROW-13150) / @ghuls
#### Related issues:
- [[C++] Allow automatic String -> LargeString promotions when concatenating tables](https://github.com/apache/arrow/issues/23539) (relates to)
- [[C++][Python] Large strings cause ArrowInvalid: offset overflow while concatenating arrays](https://github.com/apache/arrow/issues/33049) (is related to)

**Note**: *This issue was originally created as [ARROW-13150](https://issues.apache.org/jira/browse/ARROW-13150). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python ChunkedArray.combine_chunks and Table.combine_chunks entry points shown in the traceback, then compare how they reach concat_arrays. Reproduce the reported 4.0.0 case with the table and column chunk counts and sizes; done means their behavior is consistent without the reported offset overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.