NVIDIA / NVIDIA/cudf

[FEA]: Make it easier to use KvikIO from dask-cudf

Open
#19,836 0 comments 0 reactions 0 assignees View on GitHub
dask feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

Currently, using kvikio for remote IO in dask-cudf requires setting a process-global option *on each worker* (using, e.g. a WorkerPlugin):

```python
import cudf
from distributed.diagnostics.plugin import WorkerPlugin

class KvikioPlugin(WorkerPlugin):
def setup(self, worker):
cudf.set_option("kvikio_remote_io", True)

client = ... # setup up cluster / client
plugin = KvikioPlugin()
client.regsiter_worker_plugin(plugin)
```

**Describe the solution you'd like**

This is verbose and easy to get wrong. It'd be nicer if this could be controlled by either
a Dask configuration option, the `cudf` option on the client (rather than all the workers), or a keyword.

```
dask.config.set({"cudf.kvikio_remote_io": True})

df = dd.read_parquet("s3://...)

df.compute() # uses KvikIO for remote IO.
```

However its done, the fact that KvikIO should be used for Remote IO should somehow be encoded task graph, so that no special configuration is needed on the worker.

**Describe alternatives you've considered**

The worker plugin shown above is the current workaround.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.