dask / dask/distributed

scheduler.json to store additional connection arguments to the scheduler

Open
#4,877 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently, enabling Infiniband support for a Dask cluster using UCX requires that the workers and client be configured w/ the same connection arguments as the scheduler. As an example of a realistic configuration in a CUDA environment, a user might need to specify options to enable `tcp-over-ucx`, `nvlink`, `infiniband` and potentially `rdmacm`. In addition, they might need to manually specify a value for `UCX_MAX_RNDV_RAILS`. These options currently need to be specified for the scheduler, workers, and the client, which becomes painful for users who might be working to test multiple different configurations and might not know to specify one or more of them.

The more I think through the problem, I can't think of a reason why the values for these arguments would ever differ between a client/worker and the scheduler. It would make the deployment much more straightforward and alleviate a lot of the pain by storing these configuration arguments with the `scheduler.json` file.

Tagging @rlratzel and @pentschev for awareness and further thoughts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.