NVIDIA / NVIDIA/cudf

[FEA] Faster dataframe to cupy conversion when dataframe is a single allocation

Open
#12,928 4 comments 0 reactions 0 assignees View on GitHub
0 - Backlog feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

When we convert a dataframe to a cupy array, we iterate over each column (as they’re independent allocations) and assign each one to a column in an empty matrix. This means it can be slow for thousands or millions of small columns.

In a select set of circumstances, all of the columns in a DataFrame may be part of a single, contiguous allocation of memory. One scenario in which this can occur is after a call to transpose. It would be nice if, in this scenario, we didn't need to iterate over every column when converting to a cupy array.

A real-world example of when this can matter is if a user is trying to run a dot product after a calling transpose. Because of the bottleneck, we're slower than pandas by quite a bit.

```python
import cudf
import cupy as cp
import pandas as pd

nrows = 10000
ncols = 4

gdf = cudf.DataFrame(cp.random.randint(0, 1000, size=(nrows, ncols)))
pdf = gdf.to_pandas()

%time gdf.T.dot(gdf)
%time pdf.T.dot(pdf)
CPU times: user 1.52 s, sys: 3.96 ms, total: 1.53 s
Wall time: 1.53 s
CPU times: user 912 µs, sys: 41 µs, total: 953 µs
Wall time: 855 µs
```

If we were to do any special casing here, we'd want to closely evaluate any impact on performance for the more general case, as the dataframe to cupy codepath is used across the board.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.