NVIDIA / NVIDIA/cudf

[DOC]: Avoid UserWarning in 10 minute intro

Open
#19,396 0 comments 0 reactions 1 assignee Claimed by @TomAugspurger View on GitHub
doc Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Report incorrect documentation

**Location of incorrect documentation**

https://docs.rapids.ai/api/cudf/nightly/user_guide/10min/#persisting-data

**Describe the problems or issues found in the documentation**

That includes a UserWarning from dask.dataframe because we create a large in-memory dataframe and send it to the worker, rather than creating / loading data on the worker:

```
ddf2 = ddf2.persist()
ddf2
```

which prints

```
/opt/conda/envs/docs/lib/python3.13/site-packages/distributed/client.py:3363: UserWarning: Sending large graph of size 152.59 MiB.
This may cause some slowdown.
Consider loading the data with Dask directly
or using futures or delayed objects to embed the data into the graph without repetition.
See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.
warnings.warn(
```

**Steps taken to verify documentation is incorrect**
List any steps you have taken:

**Suggested fix for documentation**

We can use `dask.dataframe.from_map` to create the data on the worker. We need a helper function to get the offset right:

```python
nrows = 10000000
npartitions = 16
nrows_per_partition = nrows // npartitions
offsets = [i * nrows_per_partition for i in range(npartitions)]

def make_partition(offset, nrows_per_partition):
values = cp.arange(nrows_per_partition) + offset
return cudf.DataFrame({"a": values, "b": values}, index=values)

ddf2 = dd.from_map(make_partition, offsets, nrows_per_partition=nrows_per_partition)
ddf2["c"] = ddf2["a"] + 5
```

There's a second one later that uses more partitions. Repeat with a different `npartitions` / `nrows_per_partition`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.