[FEA]: Make it easier to use KvikIO from dask-cudf
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Currently, using kvikio for remote IO in dask-cudf requires setting a process-global option *on each worker* (using, e.g. a WorkerPlugin):
```python
import cudf
from distributed.diagnostics.plugin import WorkerPlugin
class KvikioPlugin(WorkerPlugin):
def setup(self, worker):
cudf.set_option("kvikio_remote_io", True)
client = ... # setup up cluster / client
plugin = KvikioPlugin()
client.regsiter_worker_plugin(plugin)
```
**Describe the solution you'd like**
This is verbose and easy to get wrong. It'd be nicer if this could be controlled by either
a Dask configuration option, the `cudf` option on the client (rather than all the workers), or a keyword.
```
dask.config.set({"cudf.kvikio_remote_io": True})
df = dd.read_parquet("s3://...)
df.compute() # uses KvikIO for remote IO.
```
However its done, the fact that KvikIO should be used for Remote IO should somehow be encoded task graph, so that no special configuration is needed on the worker.
**Describe alternatives you've considered**
The worker plugin shown above is the current workaround.
Contributor guide
Assessment
This issue has not been assessed yet.