apache / apache/arrow

[Python] Pandas dataframe.loc[bool_array] causes "offset overflow while concatenating arrays"

Open
#41,890 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Tested with :

- pyarrow 11, pandas 1.5
- pyarrow 16, pandas 2.0

This Python code reproduces the problem:

```
import random
import pandas as pd

NUM_ROWS = 1_000_000
MAX_WORDS_PER_SENTENCE = 1000
NUM_PRE_DEFINED_SENTENCES = 1000

WORDS = """
story where fish and children are alarmed when thing one and thing two
and the cat in the big red hat wreck the house while mother is away
""".split()

def sentence(num_words):
return " ".join(random.choices(WORDS, k=num_words))

PRE_DEFINED_SENTENCES = [sentence(random.randint(1, MAX_WORDS_PER_SENTENCE)) for s in range(NUM_PRE_DEFINED_SENTENCES)]

text_list = random.choices(PRE_DEFINED_SENTENCES, k=NUM_ROWS)

df = pd.DataFrame({"text": text_list})
df["len"] = df.text.str.len()
print(df)
print("\nMemory usage:")
print(df.memory_usage(deep=True))

df.text = df.text.astype("string[pyarrow]")

contains_fish = df.text.str.contains("fish")

df_fish = df.loc[contains_fish]
```

This is the resulting stack trace:

```
Traceback (most recent call last):
File "test_arrow.py", line 30, in
df_fish = df.loc[contains_fish]
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
return self._getitem_axis(maybe_callable, axis=axis)
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1292, in _getitem_axis
return self._getbool_axis(key, axis=axis)
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1093, in _getbool_axis
return self.obj._take_with_is_copy(inds, axis=axis)
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/generic.py", line 3902, in _take_with_is_copy
result = self._take(indices=indices, axis=axis)
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/generic.py", line 3886, in _take
new_data = self._mgr.take(
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 978, in take
return self.reindex_indexer(
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 751, in reindex_indexer
new_blocks = [
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 752, in
blk.take_nd(
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1775, in take_nd
new_values = self.values.take(indexer, fill_value=fill_value, allow_fill=True)
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/arrays/arrow/array.py", line 725, in take
return type(self)(self._data.take(indices))
File "pyarrow/table.pxi", line 1001, in pyarrow.lib.ChunkedArray.take
File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pyarrow/compute.py", line 473, in take
return call_function('take', [data, indices], options, memory_pool)
File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.