NVIDIA / NVIDIA/cudf

[BUG] Unable to update array column for subset of rows

Open
#15,233 2 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Unable to update values of a dataframe for a column of type array for a subset of rows via `.loc` with a `list[list]`.

Tested with versions: 24.02.02 & nightly 24.04.00a508

Updating the values for all rows works:
```py
df['a'] = [[0,0,0], [9, 10, 11], [20, 21, 22]]
```
however updating a subset of rows with a bool mask fails:
```py
df.loc[mask, 'a'] = new_values
```

I believe at least part of the problem is (python/cudf/cudf/core/column/lists.py) ~line 90:
```
def __setitem__(self, key, value):
if isinstance(value, list):
value = cudf.Scalar(value)
if isinstance(value, cudf.Scalar):
if value.dtype != self.dtype:
raise TypeError("list nesting level mismatch")
```
When the column is `ListDtype(int64)` and the incoming values look like `[[9, 10, 11], [20, 21, 22]]`
It will create a `cudf.Scalar` which will have a dtype of `ListDtype(ListDtype(int64))` .

Causing the `value.dtype != self.dtype` to fail.

**Steps/Code to reproduce bug**
```py
import os

import cupy
import numpy
import pandas as pd
import cudf

data = {'apple': ['pie', 'cake', 'candy'], 'a': [[3,2,1], [4,5,6], [8,7,9]], 'b': [10, 20, 30]}

if os.environ.get('USE_PANDAS') != None:
df = pd.DataFrame(data)
else:
df = cudf.DataFrame(data)

mask = [False, True, True]
new_values = [[9, 10, 11], [20, 21, 22]]

# Other datatypes work
df.loc[mask, 'b'] = [25, 35]
df['b'].loc[mask] = [26, 36]
print(df)

print(f"array column type: {repr(df['a'].dtype)}", flush=True)

try:
df.loc[mask, 'a'] = new_values
except ValueError as e:
print(f"Encoundered error setting new values ({e}) trying another way")

try:
df['a'].loc[mask] = new_values
except TypeError as e:
print(f"Encoundered error setting new values ({e})")

print(df)
```

**Expected behavior**
Update the values

**Environment overview (please complete the following information)**
- Environment location: [Bare-metal]
- Method of cuDF install: [conda]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.