intake / intake/akimbo

BUG: iterrows() on an awkward pandas column with equal-length rows results in a ValueError

Open
#55 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
52
Forks
6
PR merge metrics
No merged PRs in 30d

Description

### Reproducible Example

```python
import awkward as ak
import awkward_pandas as akpd
import pandas as pd

# numbers = [[1, 2, 3], [4, 5], [6]] # no error
numbers = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] # error
letters = ["A", "B", "C"]

numbers_ak = ak.from_iter(numbers)
numbers_akpd = akpd.from_awkward(numbers_ak)

df = pd.DataFrame({"letters": letters, "numbers": numbers_akpd})

for idx, row in df.iterrows():
print(f"{idx} - {row['letters']}, {row['numbers']}")

```

```python-traceback
File .venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2253, in EABackedBlock.get_values(self, dtype)
[2251](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2251) values = values.astype(object)
[2252](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2252) # TODO(EA2D): reshape not needed with 2D EAs
-> [2253](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2253) return np.asarray(values).reshape(self.shape)

ValueError: cannot reshape array of size 9 into shape
```

### Issue Description
I reported this issue at the Pandas repository, but they referred me to here first to verify that it is not an error with `awkward_pandas`. (see [Pandas issue 58927](https://github.com/pandas-dev/pandas/issues/58927))

---

When calling `iterrows()` on a DataFrame which contains an awkward array as a column, a ValueError occurs (see stacktrace example). This error only occurs when all rows of the awkward array are of equal length. In this case the calls to `values.astype(object)` and/or `np.asarray(values)` in the `get_values` function in the `pandas/core/internals/blocks.py` module result in a 2D array, instead of a 1D array with nested lists.
When the awkward array is actually jagged, the call results in the correct format of the array (see commented line in code example) and `iterrows()` works as intended.

### Expected Behavior

I would expect `iterrows()` to iterate over the DataFrame rows without throwing an error, but instead returning a Series with the value of the awkward array at the index of the row set correctly.

### Installed Versions
```
awkward 2.6.5
awkward_pandas 2023.8.0
numpy 1.26.3
pandas 2.2.0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.