NVIDIA / NVIDIA/cudf

[BUG] dataframe reindex NaN

Open
#8,801 14 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
cuDF DataFrame is not reindexed as intented. Using a multiindex, it ends up in NaNs.

**Steps/Code to reproduce bug**
```
data_df = cudf.read_parquet('../input/optiver-realized-volatility-prediction/book_train.parquet/stock_id=0/c439ef22282f412ba39e9137a3fdabac.parquet')
offsets = data_df.groupby(['time_id'], as_index=False).agg({'seconds_in_bucket':'min'}).reset_index(drop=True)
offsets.columns = ['time_id', 'offset']
data_df = cudf.merge(data_df, offsets, on = ['time_id'], how = 'left')
data_df.seconds_in_bucket = data_df.seconds_in_bucket - data_df.offset
# MultiIndex.from_product uses pandas in the background
# That's why we need to transform the data into pd dataframe
data_df = data_df.set_index(['time_id', 'seconds_in_bucket'])
columns = [col for col in data_df.columns.values]
data_df = data_df.reindex(cudf.MultiIndex.from_product([data_df.to_pandas().index.levels[0], np.arange(0,600)], names = ['time_id', 'seconds_in_bucket']), columns=columns).fillna(method='ffill')
data_df = cudf.DataFrame(data_df.reset_index())
```
https://www.kaggle.com/medali1992/optiver-train-dataset?scriptVersionId=68637709

This workaround works:
```
indices = cudf.MultiIndex.from_product([data_df.to_pandas().index.levels[0], np.arange(0,600)], names = ['time_id', 'seconds_in_bucket'])
data_df = cudf.DataFrame().set_index(indices).join(data_df, how="left").fillna(method='ffill').reset_index(drop=True)
```

**Expected behavior**
I expect reindex function to get the correct values instead of NaNs as in pandas.

**Environment overview (please complete the following information)**
Kaggle GPU Docker, RAPIDS 21.06

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.