Poor scaling for larger query/loc with cudf Dataframe[QST]
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
I am trying to extract the some columns of a cudf dataframe
```
%%time
cudf_file = "/work/ska/cudfoutput/lofar30MHz1_t201806301100_SBL153.parquet"
df = cudf.read_parquet(cudf_file, columns =['TIME','ANTENNA1','ANTENNA2','FLAG','DATA'])
df3 = df.loc[df['TIME'].isin(unique_time)]
for t in unique_time:
df2 = df3[df3.TIME == t]
beam_id_0_cp = cp.asarray(df2['ANTENNA1'])
beam_id_1_cp = cp.asarray(df2['ANTENNA2'])
dfflag = df2['FLAG']
data_flag_cp = cp.asarray(dfflag.list.leaves).reshape(len(dfflag),len(dfflag.iloc[0]),len(dfflag.iloc[0][0]))
dfdata = df2['DATA']
data_cp = cp.asarray(dfdata.list.leaves, dtype=np.float64).reshape(len(dfdata),len(dfdata.iloc[0]),len(dfdata.iloc[0][0])).view(np.complex128)
```
The scaling becomes worse if we have a bigger loop (>1000 steps), for smaller timesteps it is the faster but as we go higher the scaling is becoming an issue. However, I am wondering why as I am not reading the dataframe inside the loop. Is there a better way to do this? I have tried groupby as well and the results are similar
Contributor guide
Assessment
This issue has not been assessed yet.