NVIDIA / NVIDIA/cudf

Poor scaling for larger query/loc with cudf Dataframe[QST]

Open
#12,286 2 comments 0 reactions 0 assignees View on GitHub
0 - Waiting on Author Python question
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

I am trying to extract the some columns of a cudf dataframe

```
%%time
cudf_file = "/work/ska/cudfoutput/lofar30MHz1_t201806301100_SBL153.parquet"
df = cudf.read_parquet(cudf_file, columns =['TIME','ANTENNA1','ANTENNA2','FLAG','DATA'])
df3 = df.loc[df['TIME'].isin(unique_time)]
for t in unique_time:
df2 = df3[df3.TIME == t]
beam_id_0_cp = cp.asarray(df2['ANTENNA1'])
beam_id_1_cp = cp.asarray(df2['ANTENNA2'])
dfflag = df2['FLAG']
data_flag_cp = cp.asarray(dfflag.list.leaves).reshape(len(dfflag),len(dfflag.iloc[0]),len(dfflag.iloc[0][0]))
dfdata = df2['DATA']
data_cp = cp.asarray(dfdata.list.leaves, dtype=np.float64).reshape(len(dfdata),len(dfdata.iloc[0]),len(dfdata.iloc[0][0])).view(np.complex128)
```

The scaling becomes worse if we have a bigger loop (>1000 steps), for smaller timesteps it is the faster but as we go higher the scaling is becoming an issue. However, I am wondering why as I am not reading the dataframe inside the loop. Is there a better way to do this? I have tried groupby as well and the results are similar

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.