[BUG] Unable to update array column for subset of rows
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Unable to update values of a dataframe for a column of type array for a subset of rows via `.loc` with a `list[list]`.
Tested with versions: 24.02.02 & nightly 24.04.00a508
Updating the values for all rows works:
```py
df['a'] = [[0,0,0], [9, 10, 11], [20, 21, 22]]
```
however updating a subset of rows with a bool mask fails:
```py
df.loc[mask, 'a'] = new_values
```
I believe at least part of the problem is (python/cudf/cudf/core/column/lists.py) ~line 90:
```
def __setitem__(self, key, value):
if isinstance(value, list):
value = cudf.Scalar(value)
if isinstance(value, cudf.Scalar):
if value.dtype != self.dtype:
raise TypeError("list nesting level mismatch")
```
When the column is `ListDtype(int64)` and the incoming values look like `[[9, 10, 11], [20, 21, 22]]`
It will create a `cudf.Scalar` which will have a dtype of `ListDtype(ListDtype(int64))` .
Causing the `value.dtype != self.dtype` to fail.
**Steps/Code to reproduce bug**
```py
import os
import cupy
import numpy
import pandas as pd
import cudf
data = {'apple': ['pie', 'cake', 'candy'], 'a': [[3,2,1], [4,5,6], [8,7,9]], 'b': [10, 20, 30]}
if os.environ.get('USE_PANDAS') != None:
df = pd.DataFrame(data)
else:
df = cudf.DataFrame(data)
mask = [False, True, True]
new_values = [[9, 10, 11], [20, 21, 22]]
# Other datatypes work
df.loc[mask, 'b'] = [25, 35]
df['b'].loc[mask] = [26, 36]
print(df)
print(f"array column type: {repr(df['a'].dtype)}", flush=True)
try:
df.loc[mask, 'a'] = new_values
except ValueError as e:
print(f"Encoundered error setting new values ({e}) trying another way")
try:
df['a'].loc[mask] = new_values
except TypeError as e:
print(f"Encoundered error setting new values ({e})")
print(df)
```
**Expected behavior**
Update the values
**Environment overview (please complete the following information)**
- Environment location: [Bare-metal]
- Method of cuDF install: [conda]
Contributor guide
Assessment
This issue has not been assessed yet.