BUG: iterrows() on an awkward pandas column with equal-length rows results in a ValueError
- Dominant language
- Python
- Stars
- 52
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
### Reproducible Example
```python
import awkward as ak
import awkward_pandas as akpd
import pandas as pd
# numbers = [[1, 2, 3], [4, 5], [6]] # no error
numbers = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] # error
letters = ["A", "B", "C"]
numbers_ak = ak.from_iter(numbers)
numbers_akpd = akpd.from_awkward(numbers_ak)
df = pd.DataFrame({"letters": letters, "numbers": numbers_akpd})
for idx, row in df.iterrows():
print(f"{idx} - {row['letters']}, {row['numbers']}")
```
```python-traceback
File .venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2253, in EABackedBlock.get_values(self, dtype)
[2251](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2251) values = values.astype(object)
[2252](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2252) # TODO(EA2D): reshape not needed with 2D EAs
-> [2253](.venv/lib/python3.11/site-packages/pandas/core/internals/blocks.py:2253) return np.asarray(values).reshape(self.shape)
ValueError: cannot reshape array of size 9 into shape
```
### Issue Description
I reported this issue at the Pandas repository, but they referred me to here first to verify that it is not an error with `awkward_pandas`. (see [Pandas issue 58927](https://github.com/pandas-dev/pandas/issues/58927))
---
When calling `iterrows()` on a DataFrame which contains an awkward array as a column, a ValueError occurs (see stacktrace example). This error only occurs when all rows of the awkward array are of equal length. In this case the calls to `values.astype(object)` and/or `np.asarray(values)` in the `get_values` function in the `pandas/core/internals/blocks.py` module result in a 2D array, instead of a 1D array with nested lists.
When the awkward array is actually jagged, the call results in the correct format of the array (see commented line in code example) and `iterrows()` works as intended.
### Expected Behavior
I would expect `iterrows()` to iterate over the DataFrame rows without throwing an error, but instead returning a Series with the value of the awkward array at the index of the row set correctly.
### Installed Versions
```
awkward 2.6.5
awkward_pandas 2023.8.0
numpy 1.26.3
pandas 2.2.0
```
Contributor guide
Assessment
This issue has not been assessed yet.