Standard workflow for using ucx/ucx-py to speed up datatransfer between nodes/processes
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I got lot's of information pieces that ucx/ucx-py could speed up the data transfer between not only GPU-GPU communication but also CPU-CPU communication at different nodes/workers on the framework of dask/distributed. However, I can't get a full document about how to do this and what to be correctly configured.
I tried the code as below, but no speed up could be abserved.
(1)The configure for ucxpy is :
```
#.bashrc file
export UCX_MEMTYPE_CACHE=n UCX_TLS=rc,tcp,sockcm UCX_SOCKADDR_TLS_PRIORITY=sockcm
export UCXPY_IFNAME="ib0"
```
(2) then , i start the scheduler with:`dask-scheduler --protocol ucx --interface ib0`
(3) then , i start the worker with : `opt/anaconda37/bin/python3.7 -m distributed.cli.dask_worker ${scheduler_address} --nthreads 15 --nprocs 1 --memory-limit 400.00GB --nanny --death-timeout 60 --local-directory /scratch/qsong --name com --interface ib0` at different nodes of the cluster, where $(scheduler_address) is` ucx:\\ ......` like address got from last step. Besides,the ENV value at different nodes is ensured to be the same with step 1
(4) then, in the main.py, I use: `client = Client('ucx://172.16.10.12:8786')` to start the calculation.
The data could be transfered from main.py to different nodes safely, but no speed-up could be observed.
Is there any suggestion ?
@mrocklin
Contributor guide
Assessment
This issue has not been assessed yet.