rapidsai / rapidsai/deployment

ec2-multi: Dask EC2 cluster example timing out

Open
#455 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug cloud/aws help wanted
Dominant language
Jupyter Notebook
Stars
15
Forks
41
Avg merge
1d 8h
Merged PRs (30d)
6

Description

Description

For https://github.com/rapidsai/deployment/issues/434, I tried running the steps at https://docs.rapids.ai/deployment/nightly/cloud/aws/ec2-multi/ using the 24.10 version of RAPIDS (which pulls in dask==2024.9.0 and distributed==2024.9.0).

That example uses dask_cloudprovider.aws.EC2Cluster to create a cluster of EC2 instances, then encourages users to try creating a cudf DataFrame, distributing it over the cluster, and operating on it in parallel (see "Reproducible Example" below).

I saw EC2Cluster() successfully create a cluster of EC2 instances (1 scheduler and 3 workers), but the first time I tried to call .compute() on a dask_cudf.DataFrame to get a result back from the cluster, that operation hung indefinitely (did not time out after 15 minutes).

Reproducible Example

I created an EC2 instances using the "NVIDIA GPU-Optimized AMI", following https://docs.rapids.ai/deployment/nightly/cloud/aws/ec2/.

Then ran the following:

docker run \
    --rm \
    --gpus all \
    --network host \
    --env AWS_PROFILE=oct2024 \
    --entrypoint="" \
    -it rapidsai/notebooks:24.10a-cuda12.5-py3.11 \
    bash

Created a file ~/.aws/credentials in the container, with an [oct2024] profile.

Installed dask-cloudprovider, with the same version as distributed and dask.

conda install --yes -c rapidsai-nightly -c conda-forge \
    'dask-cloudprovider==2024.9.0'

Tried running the example code.

from dask_cloudprovider.aws import EC2Cluster
from dask.distributed import Client
import cudf
import dask_cudf

cluster = EC2Cluster(
    instance_type="g4dn.4xlarge",
    docker_image="rapidsai/base:24.10a-cuda12.5-py3.11",
    worker_class="dask_cuda.CUDAWorker",
    worker_options={"rmm-managed-memory": True},
    docker_args="--shm-size=256m",
    n_workers=3,
    security=False,
    availability_zone="us-east-2a",
    region="us-east-2",
)

client = Client(cluster)

df = dask_cudf.from_cudf(cudf.datasets.timeseries(), npartitions=2)

# hangs on this line
df.x.mean().compute()

This warning is emitted immediately

/opt/conda/lib/python3.11/site-packages/distributed/client.py:3361: UserWarning: Sending large graph of size 39.55 MiB.
This may cause some slowdown.
Consider loading the data with Dask directly
or using futures or delayed objects to embed the data into the graph without repetition.
See also https://docs.dask.org/en/stable/best-practices.html#load-data-with-dask for more information.
warnings.warn(

And then that .mean().compute() does not complete (after 15+ minutes).

output of 'conda info' (click me) 
     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/rapids/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 24.9.1
    conda-build version : not installed
         python version : 3.11.10.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=cascadelake
                          __conda=24.9.1=0
                          __cuda=12.5=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/rapidsai-nightly/linux-64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/dask/label/dev/linux-64
                          https://conda.anaconda.org/dask/label/dev/noarch
                          https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
          package cache : /opt/conda/pkgs
                          /home/rapids/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/rapids/.conda/envs
               platform : linux-64
             user-agent : conda/24.9.1 requests/2.32.3 CPython/3.11.10 Linux/6.5.0-1018-aws ubuntu/22.04.5 glibc/2.35 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.10
                UID:GID : 1001:1000
             netrc file : None
           offline mode : False
output of 'conda env export' (click me) 
name: base
channels:
  - rapidsai-nightly
  - dask/label/dev
  - pytorch
  - conda-forge
  - nvidia
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - affine=2.4.0=pyhd8ed1ab_0
  - aiobotocore=2.15.1=pyhd8ed1ab_0
  - aiohappyeyeballs=2.4.3=pyhd8ed1ab_0
  - aiohttp=3.10.9=py311h9ecbd09_0
  - aioitertools=0.12.0=pyhd8ed1ab_0
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - alsa-lib=1.2.12=h4ab18f5_0
  - anyio=4.6.0=pyhd8ed1ab_1
  - aom=3.9.1=hac33072_0
  - archspec=0.2.3=pyhd8ed1ab_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py311h9ecbd09_5
  - arrow=1.3.0=pyhd8ed1ab_0
  - asttokens=2.4.1=pyhd8ed1ab_0
  - async-lru=2.0.4=pyhd8ed1ab_0
  - attrs=24.2.0=pyh71513ae_0
  - aws-c-auth=0.7.22=hf36ad8f_6
  - aws-c-cal=0.6.15=h816f305_1
  - aws-c-common=0.9.23=h4ab18f5_0
  - aws-c-compression=0.2.18=he027950_7
  - aws-c-event-stream=0.4.2=hb72ac1a_14
  - aws-c-http=0.8.2=h75ac8c9_3
  - aws-c-io=0.14.9=hd3d3696_3
  - aws-c-mqtt=0.10.4=hb0abfc5_7
  - aws-c-s3=0.5.10=h44b787d_4
  - aws-c-sdkutils=0.1.16=he027950_3
  - aws-checksums=0.1.18=he027950_7
  - aws-crt-cpp=0.26.12=he940a02_1
  - aws-sdk-cpp=1.11.329=h0f5bab0_6
  - awscli=2.18.1=py311h38be061_0
  - awscrt=0.20.12=py311h1684233_1
  - azure-core-cpp=1.13.0=h935415a_0
  - azure-identity-cpp=1.9.0=hd126650_0
  - azure-storage-blobs-cpp=12.13.0=h1d30c4a_0
  - azure-storage-common-cpp=12.8.0=ha3822c6_0
  - azure-storage-files-datalake-cpp=12.12.0=h0f25b8a_0
  - babel=2.14.0=pyhd8ed1ab_0
  - beautifulsoup4=4.12.3=pyha770c72_0
  - bleach=6.1.0=pyhd8ed1ab_0
  - blosc=1.21.6=hef167b5_0
  - bokeh=3.5.2=pyhd8ed1ab_0
  - boltons=24.0.0=pyhd8ed1ab_0
  - botocore=1.35.23=pyge310_1234567_0
  - branca=0.7.2=pyhd8ed1ab_0
  - brotli=1.1.0=hb9d3cd8_2
  - brotli-bin=1.1.0=hb9d3cd8_2
  - brotli-python=1.1.0=py311hfdbb021_2
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=h4bc722e_7
  - c-ares=1.33.1=heb4867d_0
  - c-blosc2=2.15.1=hc57e6cf_0
  - ca-certificates=2024.8.30=hbcca054_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cachetools=5.5.0=pyhd8ed1ab_0
  - cairo=1.18.0=hebfffa5_3
  - certifi=2024.8.30=pyhd8ed1ab_0
  - cffi=1.17.1=py311hf29c0ef_0
  - charls=2.4.2=h59595ed_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - click-plugins=1.1.1=py_0
  - cligj=0.7.2=pyhd8ed1ab_1
  - cloudpickle=3.0.0=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - colorcet=3.1.0=pyhd8ed1ab_0
  - comm=0.2.2=pyhd8ed1ab_0
  - conda=24.9.1=py311h38be061_0
  - conda-libmamba-solver=24.9.0=pyhd8ed1ab_0
  - conda-package-handling=2.3.0=pyh7900ff3_0
  - conda-package-streaming=0.10.0=pyhd8ed1ab_0
  - contourpy=1.3.0=py311hd18a35c_2
  - coverage=7.6.1=py311h9ecbd09_1
  - cryptography=40.0.2=py311h9b4c7bb_0
  - cucim=24.10.00a21=cuda12_py311_241008_g289e340_21
  - cuda-cccl_linux-64=12.5.39=ha770c72_0
  - cuda-crt-dev_linux-64=12.5.82=ha770c72_0
  - cuda-crt-tools=12.5.82=ha770c72_0
  - cuda-cudart=12.5.82=he02047a_0
  - cuda-cudart-dev=12.5.82=he02047a_0
  - cuda-cudart-dev_linux-64=12.5.82=h85509e4_0
  - cuda-cudart-static=12.5.82=he02047a_0
  - cuda-cudart-static_linux-64=12.5.82=h85509e4_0
  - cuda-cudart_linux-64=12.5.82=h85509e4_0
  - cuda-nvcc-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvcc-impl=12.5.82=hd3aeb46_0
  - cuda-nvcc-tools=12.5.82=hd3aeb46_0
  - cuda-nvrtc=12.5.82=he02047a_0
  - cuda-nvvm-dev_linux-64=12.5.82=ha770c72_0
  - cuda-nvvm-impl=12.5.82=h59595ed_0
  - cuda-nvvm-tools=12.5.82=h59595ed_0
  - cuda-profiler-api=12.5.39=ha770c72_0
  - cuda-python=12.6.0=py311h817de4b_0
  - cuda-version=12.5=hd4f0392_3
  - cudf=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cudf_kafka=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cugraph=24.10.00a96=cuda12_py311_241008_gacf502987_96
  - cuml=24.10.00a67=cuda12_py311_241008_g70fe52684_67
  - cuproj=24.10.00a48=cuda12_py311_241008_gcc8fd60b_48
  - cupy=13.3.0=py311h1c6efab_0
  - cupy-core=13.3.0=py311h95322c3_0
  - curl=8.10.1=hbbe4b11_0
  - cuspatial=24.10.00a48=cuda12_py311_241008_gcc8fd60b_48
  - custreamz=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - cuvs=24.10.00a92=cuda12_py311_241008_gc14b879_92
  - cuxfilter=24.10.00a25=cuda12_py311_241008_g0cd1568_25
  - cycler=0.12.1=pyhd8ed1ab_0
  - cyrus-sasl=2.1.27=h54b06d7_7
  - cython=3.0.11=py311h55d416d_3
  - cytoolz=1.0.0=py311h9ecbd09_1
  - dask=2024.9.0=pyhd8ed1ab_0
  - dask-cloudprovider=2024.9.1=pyhd8ed1ab_0
  - dask-core=2024.9.0=pyhd8ed1ab_0
  - dask-cuda=24.10.00a23=py311_241008_g1c84a6a_23
  - dask-cudf=24.10.00a401=cuda12_py311_241007_g8a9df040e1_401
  - dask-expr=1.1.14=pyhd8ed1ab_0
  - dask-glm=0.3.2=pyhd8ed1ab_0
  - dask-labextension=7.0.0=pyhd8ed1ab_0
  - dask-ml=2023.3.24=pyhd8ed1ab_1
  - datashader=0.16.3=pyhd8ed1ab_0
  - dav1d=1.2.1=hd590300_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.8.6=py311hfdbb021_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - distributed=2024.9.0=pyhd8ed1ab_0
  - distributed-ucxx=0.40.00a=py3.11_241008_g1d169a4_33
  - distro=1.8.0=pyhd8ed1ab_0
  - dlpack=0.8=h59595ed_3
  - docutils=0.19=py311h38be061_1
  - double-conversion=3.3.0=h59595ed_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - exceptiongroup=1.2.2=pyhd8ed1ab_0
  - execnet=2.1.1=pyhd8ed1ab_0
  - executing=2.1.0=pyhd8ed1ab_0
  - expat=2.6.3=h5888daf_0
  - fastrlock=0.8.2=py311hb755f60_2
  - fmt=11.0.2=h434a139_0
  - folium=0.17.0=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_3
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.54.1=py311h9ecbd09_0
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - freexl=2.0.0=h743c826_0
  - frozendict=2.4.4=py311h9ecbd09_1
  - frozenlist=1.4.1=py311h9ecbd09_1
  - fsspec=2024.9.0=pyhff2d567_0
  - gdal=3.9.2=py311h5159542_7
  - geopandas=1.0.1=pyhd8ed1ab_1
  - geopandas-base=1.0.1=pyha770c72_1
  - geos=3.13.0=h5888daf_0
  - geotiff=1.7.3=h77b800c_3
  - gettext=0.22.5=he02047a_3
  - gettext-tools=0.22.5=he02047a_3
  - gflags=2.2.2=h5888daf_1005
  - giflib=5.2.2=hd590300_0
  - glog=0.7.1=hbabe93e_0
  - graphite2=1.3.13=h59595ed_1003
  - h11=0.14.0=pyhd8ed1ab_0
  - h2=4.1.0=pyhd8ed1ab_0
  - harfbuzz=9.0.0=hda332d3_1
  - holoviews=1.19.1=pyhd8ed1ab_0
  - hpack=4.0.0=pyh9f0ad1d_0
  - httpcore=1.0.6=pyhd8ed1ab_0
  - httpx=0.27.2=pyhd8ed1ab_0
  - hyperframe=6.0.1=pyhd8ed1ab_0
  - icu=75.1=he02047a_0
  - idna=3.10=pyhd8ed1ab_0
  - imagecodecs=2024.6.1=py311h1c1dfb1_5
  - imageio=2.35.1=pyh12aca89_0
  - importlib-metadata=8.5.0=pyha770c72_0
  - importlib_metadata=8.5.0=hd8ed1ab_0
  - importlib_resources=6.4.5=pyhd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - ipykernel=6.29.5=pyh3099207_0
  - ipython=8.17.2=pyh41d4057_0
  - ipywidgets=8.1.5=pyhd8ed1ab_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - jedi=0.19.1=pyhd8ed1ab_0
  - jinja2=3.1.4=pyhd8ed1ab_0
  - jmespath=1.0.1=pyhd8ed1ab_0
  - joblib=1.4.2=pyhd8ed1ab_0
  - json-c=0.18=h6688a6e_0
  - json5=0.9.25=pyhd8ed1ab_0
  - jsonpatch=1.33=pyhd8ed1ab_0
  - jsonpointer=3.0.0=py311h38be061_1
  - jsonschema=4.23.0=pyhd8ed1ab_0
  - jsonschema-specifications=2023.12.1=pyhd8ed1ab_0
  - jsonschema-with-format-nongpl=4.23.0=hd8ed1ab_0
  - jupyter=1.1.1=pyhd8ed1ab_0
  - jupyter-lsp=2.2.5=pyhd8ed1ab_0
  - jupyter-server-proxy=4.4.0=pyhd8ed1ab_0
  - jupyter_client=8.6.3=pyhd8ed1ab_0
  - jupyter_console=6.6.3=pyhd8ed1ab_0
  - jupyter_core=5.7.2=pyh31011fe_1
  - jupyter_events=0.10.0=pyhd8ed1ab_0
  - jupyter_server=2.14.2=pyhd8ed1ab_0
  - jupyter_server_terminals=0.5.3=pyhd8ed1ab_0
  - jupyterlab=4.2.5=pyhd8ed1ab_0
  - jupyterlab-nvdashboard=0.11.00=py_240603_gf481e0c_0
  - jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
  - jupyterlab_server=2.27.3=pyhd8ed1ab_0
  - jupyterlab_widgets=3.0.13=pyhd8ed1ab_0
  - jxrlib=1.1=hd590300_3
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.7=py311hd18a35c_0
  - krb5=1.21.3=h659f571_0
  - lazy-loader=0.4=pyhd8ed1ab_1
  - lazy_loader=0.4=pyhd8ed1ab_1
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.43=h712a8e2_1
  - lerc=4.0.0=h27087fc_0
  - libabseil=20240116.2=cxx17_he02047a_1
  - libaec=1.1.3=h59595ed_0
  - libarchive=3.7.4=hfca40fe_0
  - libarrow=16.1.0=h9102155_9_cpu
  - libarrow-acero=16.1.0=hac33072_9_cpu
  - libarrow-dataset=16.1.0=hac33072_9_cpu
  - libarrow-substrait=16.1.0=h7e0c224_9_cpu
  - libasprintf=0.22.5=he8f35ee_3
  - libasprintf-devel=0.22.5=he8f35ee_3
  - libavif16=1.1.1=h104a339_1
  - libblas=3.9.0=24_linux64_openblas
  - libbrotlicommon=1.1.0=hb9d3cd8_2
  - libbrotlidec=1.1.0=hb9d3cd8_2
  - libbrotlienc=1.1.0=hb9d3cd8_2
  - libcblas=3.9.0=24_linux64_openblas
  - libclang-cpp19.1=19.1.0=default_hb5137d0_0
  - libclang13=19.1.0=default_h9c6a7e4_0
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libcublas=12.5.3.2=he02047a_0
  - libcublas-dev=12.5.3.2=he02047a_0
  - libcucim=24.10.00a21=cuda12_241008_g289e340_21
  - libcudf=24.10.00a401=cuda12_241008_g8a9df040e1_401
  - libcudf_kafka=24.10.00a401=cuda12_241008_g8a9df040e1_401
  - libcufft=11.2.3.61=he02047a_0
  - libcufile=1.10.1.7=he02047a_0
  - libcufile-dev=1.10.1.7=he02047a_0
  - libcugraph=24.10.00a96=cuda12_241008_gacf502987_96
  - libcugraph_etl=24.10.00a96=cuda12_241008_gacf502987_96
  - libcugraphops=24.10.00a15=cuda12_241008_g0bfc7c51_15
  - libcuml=24.10.00a67=cuda12_241008_g70fe52684_67
  - libcumlprims=24.10.00a=cuda12_241008_g51defe5_9
  - libcups=2.3.3=h4637d8d_4
  - libcurand=10.3.6.82=he02047a_0
  - libcurand-dev=10.3.6.82=he02047a_0
  - libcurl=8.10.1=hbbe4b11_0
  - libcusolver=11.6.3.83=he02047a_0
  - libcusolver-dev=11.6.3.83=he02047a_0
  - libcusparse=12.5.1.3=he02047a_0
  - libcusparse-dev=12.5.1.3=he02047a_0
  - libcuspatial=24.10.00a48=cuda12_241008_gcc8fd60b_48
  - libcuvs=24.10.00a92=cuda12_241008_gc14b879_92
  - libdeflate=1.22=hb9d3cd8_0
  - libdrm=2.4.123=hb9d3cd8_0
  - libedit=3.1.20191231=he28a2e2_2
  - libegl=1.7.0=ha4b6fd6_1
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libexpat=2.6.3=h5888daf_0
  - libffi=3.4.2=h7f98852_5
  - libgcc=14.1.0=h77fa898_1
  - libgcc-ng=14.1.0=h69a702a_1
  - libgdal-core=3.9.2=hd5b9bfb_7
  - libgettextpo=0.22.5=he02047a_3
  - libgettextpo-devel=0.22.5=he02047a_3
  - libgfortran=14.1.0=h69a702a_1
  - libgfortran-ng=14.1.0=h69a702a_1
  - libgfortran5=14.1.0=hc5f4f2c_1
  - libgl=1.7.0=ha4b6fd6_1
  - libglib=2.82.1=h2ff4ddf_0
  - libglvnd=1.7.0=ha4b6fd6_1
  - libglx=1.7.0=ha4b6fd6_1
  - libgomp=14.1.0=h77fa898_1
  - libgoogle-cloud=2.25.0=h2736e30_0
  - libgoogle-cloud-storage=2.25.0=h3d9a0c8_0
  - libgrpc=1.62.2=h15f2491_0
  - libhwy=1.1.0=h00ab1b0_0
  - libiconv=1.17=hd590300_2
  - libidn2=2.3.7=hd590300_0
  - libjpeg-turbo=3.0.0=hd590300_1
  - libjxl=0.11.0=hdb8da77_1
  - libkml=1.3.0=hf539b9f_1021
  - libkvikio=24.10.00a=cuda12_241008_g5a733bd_53
  - liblapack=3.9.0=24_linux64_openblas
  - libllvm14=14.0.6=hcd5def8_4
  - libllvm19=19.1.1=ha7bfdaf_0
  - libmamba=1.5.10=hf72d635_1
  - libmambapy=1.5.10=py311h18a8eac_1
  - libnghttp2=1.58.0=h47da74e_1
  - libnl=3.10.0=h4bc722e_0
  - libnsl=2.0.1=hd590300_0
  - libntlm=1.4=h7f98852_1002
  - libnvjitlink=12.5.82=he02047a_0
  - libnvjpeg=12.3.2.81=he02047a_0
  - libopenblas=0.3.27=pthreads_hac2b453_1
  - libopengl=1.7.0=ha4b6fd6_1
  - libparquet=16.1.0=h6a7eafb_9_cpu
  - libpciaccess=0.18=hd590300_0
  - libpng=1.6.44=hadc24fc_0
  - libpq=17.0=h04577a9_2
  - libprotobuf=4.25.3=hd5b35b9_1
  - libraft=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - libraft-headers=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - libraft-headers-only=24.10.00a48=cuda12_241008_g6c4fdfb3_48
  - librdkafka=2.5.3=h95ba008_0
  - libre2-11=2023.09.01=h5a48ba9_2
  - librmm=24.10.00a42=cuda12_241008_gab6e2961_42
  - librttopo=1.1.0=h97f6797_17
  - libsodium=1.0.20=h4ab18f5_0
  - libsolv=0.7.30=h3509ff9_0
  - libspatialite=5.1.0=h1b4f908_11
  - libsqlite=3.46.1=hadc24fc_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx=14.1.0=hc0a3c3a_1
  - libstdcxx-ng=14.1.0=h4852527_1
  - libthrift=0.19.0=hb90f79a_1
  - libtiff=4.7.0=he137b08_1
  - libucxx=0.40.00a=cuda12_241008_g1d169a4_33
  - libunistring=0.9.10=h7f98852_0
  - libutf8proc=2.8.0=h166bdaf_0
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.49.0=hb9d3cd8_0
  - libwebp-base=1.4.0=hd590300_0
  - libwholegraph=24.10.00a20=cuda12_241008_gf8ad9a1_20
  - libxcb=1.17.0=h8a09558_0
  - libxcrypt=4.4.36=hd590300_1
  - libxgboost=2.1.1=rapidsai_h01f03eb_5
  - libxkbcommon=1.7.0=h2c5496b_1
  - libxml2=2.12.7=he7c6b58_4
  - libxslt=1.1.39=h76b75d6_0
  - libzlib=1.3.1=hb9d3cd8_2
  - libzopfli=1.0.3=h9c3ff4c_0
  - linkify-it-py=2.0.3=pyhd8ed1ab_0
  - llvmlite=0.43.0=py311h9c9ff8c_1
  - locket=1.0.0=pyhd8ed1ab_0
  - lz4=4.3.3=py311h2cbdf9a_1
  - lz4-c=1.9.4=hcb278e6_0
  - lzo=2.10=hd590300_1001
  - mamba=1.5.10=py311h3072747_1
  - mapclassify=2.8.1=pyhd8ed1ab_0
  - markdown=3.6=pyhd8ed1ab_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=3.0.0=py311h9ecbd09_0
  - matplotlib=3.9.2=py311h38be061_1
  - matplotlib-base=3.9.2=py311h2b939e6_1
  - matplotlib-inline=0.1.7=pyhd8ed1ab_0
  - mdit-py-plugins=0.4.2=pyhd8ed1ab_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - menuinst=2.1.2=py311h38be061_1
  - minizip=4.0.7=h401b404_0
  - mistune=3.0.2=pyhd8ed1ab_0
  - msgpack-python=1.1.0=py311hd18a35c_0
  - multidict=6.1.0=py311h9ecbd09_0
  - multipledispatch=0.6.0=pyhd8ed1ab_1
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=9.0.1=h266115a_1
  - mysql-libs=9.0.1=he0572af_1
  - nbclient=0.10.0=pyhd8ed1ab_0
  - nbconvert-core=7.16.4=pyhd8ed1ab_1
  - nbformat=5.10.4=pyhd8ed1ab_0
  - nccl=2.23.4.1=h52f6c39_0
  - ncurses=6.5=he02047a_1
  - nest-asyncio=1.6.0=pyhd8ed1ab_0
  - networkx=3.3=pyhd8ed1ab_1
  - nodejs=22.9.0=hf235a45_0
  - notebook=7.2.2=pyhd8ed1ab_0
  - notebook-shim=0.2.4=pyhd8ed1ab_0
  - numba=0.60.0=py311h4bc866e_0
  - numpy=1.26.4=py311h64a7726_0
  - nvcomp=4.0.1=hbc370b7_0
  - nvtx=0.2.10=py311h9ecbd09_2
  - nx-cugraph=24.10.00a96=py311_241008_gacf502987_96
  - openjpeg=2.5.2=h488ebb8_0
  - openldap=2.6.8=hedd0468_0
  - openssl=3.3.2=hb9d3cd8_0
  - orc=2.0.1=h17fec99_1
  - osmnx=1.9.3=pyhd8ed1ab_0
  - overrides=7.7.0=pyhd8ed1ab_0
  - packaging=24.1=pyhd8ed1ab_0
  - pandas=2.2.2=py311h14de704_1
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - panel=1.5.2=pyhd8ed1ab_0
  - param=2.1.1=pyhff2d567_0
  - parso=0.8.4=pyhd8ed1ab_0
  - partd=1.4.2=pyhd8ed1ab_0
  - patsy=0.5.6=pyhd8ed1ab_0
  - pcre2=10.44=hba22ea6_2
  - perl=5.32.1=7_hd590300_perl5
  - pexpect=4.9.0=pyhd8ed1ab_0
  - pickleshare=0.7.5=py_1003
  - pillow=10.4.0=py311h4aec55e_1
  - pip=24.2=pyh8b19718_1
  - pixman=0.43.2=h59595ed_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - platformdirs=4.3.6=pyhd8ed1ab_0
  - pluggy=1.5.0=pyhd8ed1ab_0
  - ply=3.11=pyhd8ed1ab_2
  - proj=9.5.0=h12925eb_0
  - prometheus_client=0.21.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.38=pyha770c72_0
  - prompt_toolkit=3.0.38=hd8ed1ab_0
  - psutil=6.0.0=py311h9ecbd09_1
  - pthread-stubs=0.4=hb9d3cd8_1002
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.3=pyhd8ed1ab_0
  - py-cpuinfo=9.0.0=pyhd8ed1ab_0
  - py-xgboost=2.1.1=rapidsai_pyh199b97d_5
  - pyarrow=16.1.0=py311hbd00459_4
  - pyarrow-core=16.1.0=py311h8c3dac4_4_cpu
  - pyarrow-hotfix=0.6=pyhd8ed1ab_0
  - pybind11-abi=4=hd8ed1ab_3
  - pycosat=0.6.6=py311h459d7ec_0
  - pycparser=2.22=pyhd8ed1ab_0
  - pyct=0.5.0=pyhd8ed1ab_0
  - pydeck=0.8.0=pyhd8ed1ab_0
  - pygments=2.18.0=pyhd8ed1ab_0
  - pylibcudf=24.10.00a401=cuda12_py311_241008_g8a9df040e1_401
  - pylibcugraph=24.10.00a96=cuda12_py311_241008_gacf502987_96
  - pylibraft=24.10.00a48=cuda12_py311_241008_g6c4fdfb3_48
  - pylibwholegraph=24.10.00a20=cuda12_py311_241008_gf8ad9a1_20
  - pynvjitlink=0.3.0=py311hd269673_0
  - pynvml=11.4.1=pyhd8ed1ab_0
  - pyogrio=0.10.0=py311h5fbebbf_0
  - pyparsing=3.1.4=pyhd8ed1ab_0
  - pyproj=3.7.0=py311h0f98d5a_0
  - pyside6=6.7.3=py311h9053184_1
  - pysocks=1.7.1=pyha2e5f31_6
  - pytest=8.3.3=pyhd8ed1ab_0
  - pytest-benchmark=4.0.0=pyhd8ed1ab_0
  - pytest-cov=5.0.0=pyhd8ed1ab_0
  - pytest-xdist=3.6.1=pyhd8ed1ab_0
  - python=3.11.10=hc5c86c4_2_cpython
  - python-confluent-kafka=2.5.3=py311h9ecbd09_0
  - python-dateutil=2.9.0=pyhd8ed1ab_0
  - python-fastjsonschema=2.20.0=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python-louvain=0.16=pyhd8ed1ab_0
  - python-tzdata=2024.2=pyhd8ed1ab_0
  - python_abi=3.11=5_cp311
  - pytz=2024.2=pyhd8ed1ab_0
  - pyviz_comms=3.0.3=pyhd8ed1ab_0
  - pywavelets=1.7.0=py311h9f3472d_1
  - pyyaml=6.0.2=py311h9ecbd09_1
  - pyzmq=26.2.0=py311h7deb3e3_2
  - qhull=2020.2=h434a139_5
  - qt6-main=6.7.3=h6e8976b_1
  - raft-dask=24.10.00a48=cuda12_py311_241008_g6c4fdfb3_48
  - rapids=24.10.00a=cuda12_py311_240930_gfb695dd_11
  - rapids-dask-dependency=24.10.00a8=py_0
  - rapids-xgboost=24.10.00a=cuda12_py311_240930_gfb695dd_11
  - rasterio=1.4.1=py311hfbe26e2_0
  - rav1e=0.6.6=he8a937b_2
  - rdma-core=54.0=h5888daf_0
  - re2=2023.09.01=h7f4b329_2
  - readline=8.2=h8228510_1
  - referencing=0.35.1=pyhd8ed1ab_0
  - reproc=14.2.4.post0=hd590300_1
  - reproc-cpp=14.2.4.post0=h59595ed_1
  - requests=2.32.3=pyhd8ed1ab_0
  - rfc3339-validator=0.1.4=pyhd8ed1ab_0
  - rfc3986-validator=0.1.1=pyh9f0ad1d_0
  - rich=13.9.2=pyhd8ed1ab_0
  - rmm=24.10.00a42=cuda12_py311_241008_gab6e2961_42
  - rpds-py=0.20.0=py311h9e33e62_1
  - ruamel.yaml=0.17.21=py311h2582759_3
  - ruamel.yaml.clib=0.2.8=py311h459d7ec_0
  - s2n=1.4.16=he19d79f_0
  - scikit-image=0.24.0=py311h044e617_2
  - scikit-learn=1.5.0=py311he08f58d_1
  - scipy=1.14.1=py311he1f765f_0
  - seaborn=0.13.2=hd8ed1ab_2
  - seaborn-base=0.13.2=pyhd8ed1ab_2
  - send2trash=1.8.3=pyh0d859eb_0
  - setuptools=75.1.0=pyhd8ed1ab_0
  - shapely=2.0.6=py311h2fdb869_2
  - simpervisor=1.0.0=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.2.1=ha2e4443_0
  - sniffio=1.3.1=pyhd8ed1ab_0
  - snuggs=1.4.7=pyhd8ed1ab_1
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - soupsieve=2.5=pyhd8ed1ab_1
  - sparse=0.15.4=pyh267e887_1
  - spdlog=1.14.1=hed91bc2_1
  - sqlite=3.46.1=h9eae976_0
  - stack_data=0.6.2=pyhd8ed1ab_0
  - statsmodels=0.14.4=py311h9f3472d_0
  - streamz=0.6.4=pyh6c4a22f_0
  - svt-av1=2.2.1=h5888daf_0
  - tblib=3.0.0=pyhd8ed1ab_0
  - terminado=0.18.1=pyh0d859eb_0
  - threadpoolctl=3.5.0=pyhc1e730c_0
  - thriftpy2=0.5.2=py311h9ecbd09_1
  - tifffile=2024.9.20=pyhd8ed1ab_0
  - tinycss2=1.3.0=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.2=pyhd8ed1ab_0
  - toolz=1.0.0=pyhd8ed1ab_0
  - tornado=6.4.1=py311h9ecbd09_1
  - tqdm=4.66.5=pyhd8ed1ab_0
  - traitlets=5.14.3=pyhd8ed1ab_0
  - treelite=4.3.0=py311he8f9275_0
  - truststore=0.9.2=pyhd8ed1ab_0
  - types-python-dateutil=2.9.0.20241003=pyhff2d567_0
  - typing-extensions=4.12.2=hd8ed1ab_0
  - typing_extensions=4.12.2=pyha770c72_0
  - typing_utils=0.1.0=pyhd8ed1ab_0
  - tzdata=2024b=hc8b5060_0
  - uc-micro-py=1.0.3=pyhd8ed1ab_0
  - ucx=1.17.0=h05e919c_3
  - ucx-proc=1.0.0=gpu
  - ucx-py=0.40.00a15=py311_241008_gaf0bf5c_15
  - ucxx=0.40.00a=cuda12_py3.11_241008_g1d169a4_33
  - uri-template=1.3.0=pyhd8ed1ab_0
  - uriparser=0.9.8=hac33072_0
  - urllib3=1.26.19=pyhd8ed1ab_0
  - vim=9.1.0611=py311pl5321ha5a8562_1
  - wayland=1.23.1=h3e06ad9_0
  - wcwidth=0.2.13=pyhd8ed1ab_0
  - webcolors=24.8.0=pyhd8ed1ab_0
  - webencodings=0.5.1=pyhd8ed1ab_2
  - websocket-client=1.8.0=pyhd8ed1ab_0
  - wget=1.21.4=hda4d442_0
  - wheel=0.44.0=pyhd8ed1ab_0
  - widgetsnbextension=4.0.13=pyhd8ed1ab_0
  - wrapt=1.16.0=py311h9ecbd09_1
  - xarray=2024.9.0=pyhd8ed1ab_0
  - xcb-util=0.4.1=hb711507_2
  - xcb-util-cursor=0.1.5=hb9d3cd8_0
  - xcb-util-image=0.4.0=hb711507_2
  - xcb-util-keysyms=0.4.1=hb711507_0
  - xcb-util-renderutil=0.3.10=hb711507_0
  - xcb-util-wm=0.4.2=hb711507_0
  - xerces-c=3.2.5=h988505b_2
  - xgboost=2.1.1=rapidsai_pyh9f47a55_5
  - xkeyboard-config=2.43=hb9d3cd8_0
  - xorg-libice=1.1.1=hb9d3cd8_1
  - xorg-libsm=1.2.4=he73a12e_1
  - xorg-libx11=1.8.10=h4f16b4b_0
  - xorg-libxau=1.0.11=hb9d3cd8_1
  - xorg-libxcomposite=0.4.6=hb9d3cd8_2
  - xorg-libxcursor=1.2.2=hb9d3cd8_0
  - xorg-libxdamage=1.1.6=hb9d3cd8_0
  - xorg-libxdmcp=1.1.5=hb9d3cd8_0
  - xorg-libxext=1.3.6=hb9d3cd8_0
  - xorg-libxfixes=6.0.1=hb9d3cd8_0
  - xorg-libxi=1.8.2=hb9d3cd8_0
  - xorg-libxrandr=1.5.4=hb9d3cd8_0
  - xorg-libxrender=0.9.11=hb9d3cd8_1
  - xorg-libxt=1.3.0=hb9d3cd8_2
  - xorg-libxtst=1.2.5=hb9d3cd8_3
  - xorg-libxxf86vm=1.1.5=hb9d3cd8_3
  - xorg-xorgproto=2024.1=hb9d3cd8_1
  - xyzservices=2024.9.0=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yaml-cpp=0.8.0=h59595ed_0
  - yarl=1.13.1=py311h9ecbd09_0
  - zeromq=4.3.5=h3b0a872_6
  - zfp=1.0.1=h5888daf_2
  - zict=3.0.0=pyhd8ed1ab_0
  - zipp=3.20.2=pyhd8ed1ab_0
  - zlib=1.3.1=hb9d3cd8_2
  - zlib-ng=2.2.2=h5888daf_0
  - zstandard=0.23.0=py311hbc35293_1
  - zstd=1.5.6=ha6fb4c9_0
prefix: /opt/conda

Notes

Maybe related: #343

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the EC2 multi-node example at docs.rapids.ai/deployment/nightly/cloud/aws/ec2-multi/ and reproduce the provided EC2Cluster code, especially df.x.mean().compute(). Compare the cluster startup with the distributed computation and confirm the example completes instead of hanging; no repository file or test is named in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
cloud, distributed-systems, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.