dask / dask/distributed

got an unexpected keyword argument 'keys' on Summit with distributed 2023.6.1 - 2023.12.0

Open
#8,403 4 comments 0 reactions 0 assignees View on GitHub
bug needs info
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

The dask scheduler threw an exception when workers began to register with it. The error message was "got an unexpected keyword argument 'keys'". This was with code that had worked before, the only significant change was using a more recent version of Dask distributed, 2023.12.0. I back installed earlier versions going back to 2023.6.1 and got the same error message.

This is a more detailed stack trace:

```
2023-12-12 17:19:59,164 - distributed.scheduler - INFO - Receive client connection: Client-9626b5a7-993c-11ee-a4ed-70e284143e58
2023-12-12 17:19:59,435 - distributed.core - INFO - Starting established connection to tcp://10.41.0.45:53396
2023-12-12 17:20:06,751 - distributed.core - ERROR - add_worker() got an unexpected keyword argument 'keys'
Traceback (most recent call last):
File "/gpfs/alpine/csc505/proj-shared/conda_intersect/lib/python3.9/site-packages/distributed/utils.py", line 801, in wrapper
return await func(*args, **kwargs)
TypeError: add_worker() got an unexpected keyword argument 'keys'
2023-12-12 17:20:06,751 - distributed.core - ERROR - Exception while handling op register-worker
Traceback (most recent call last):
File "/gpfs/alpine/csc505/proj-shared/conda_intersect/lib/python3.9/site-packages/distributed/core.py", line 921, in _handle_comm
result = await result
File "/gpfs/alpine/csc505/proj-shared/conda_intersect/lib/python3.9/site-packages/distributed/utils.py", line 801, in wrapper
return await func(*args, **kwargs)
TypeError: add_worker() got an unexpected keyword argument 'keys'
```

I resolved this locally by reverting to Dask distributed 2023.1.1.

**Minimal Complete Verifiable Example**:

I will post salient portions of my LSF batch submission script.

```bash
#!/bin/sh
#
# Batch submission script for a test run w/ two nodes.
#
#BSUB -P csc505
#BSUB -W 00:30
#BSUB -nnodes 2
#BSUB -J 16-ea-plain-tuning
#BSUB -o out.atomai.%J
#BSUB -e err.atomai.%J
#BSUB -alloc_flags "gpudefault"
#BSUB -q debug
#BSUB -B
#BSUB -N

# make the run directory and move into it; we use the job ID to ensure we have
# unique directory to catch all output.
export RUN_DIR=$(realpath .)/${LSB_JOBID}
if [ ! -d "$RUN_DIR" ]
then
mkdir -p $RUN_DIR
fi
cd $RUN_DIR

# dask file for scheduler and workers to find each other
export SCHEDULER_FILE=${RUN_DIR}/scheduler_file.json

dask scheduler --interface ib0 --no-dashboard --no-show \
--scheduler-file $SCHEDULER_FILE > dask-scheduler.out 2>&1 &

# Give the scheduler a chance to spin up.
sleep 5

# We allocate resource sets for entire nodes and use a shell script to dole
# out dask workers per GPU.
jsrun -h $RUN_DIR \
--smpiargs="off" --nrs $NUM_NODES -e individual \
--stdio_stdout ${RUN_DIR}/worker_out.%h \
--stdio_stderr ${RUN_DIR}/worker_error.%h \
--tasks_per_rs 1 --cpu_per_rs 38 --gpu_per_rs 6 --rs_per_host 1 -b none \
--latency_priority gpu-cpu --launch_distribution cyclic \
${SCRIPTS_DIR}/start_workers.sh &
```

And this is `start_workers.sh`:

```bash
for gpu in $(seq 0 5); do
echo Setting up for GPU rank $gpu on $(hostname) ;
(env -v CUDA_VISIBLE_DEVICES=${gpu} dask worker \
--scheduler-file $SCHEDULER_FILE --local-directory /tmp \
--nthreads 1 --nworkers 1 \
--interface ib0 --no-dashboard --no-nanny --death-timeout 600) &
sleep 2 ;
done

# If we don't wait, then this script exits, killing off the workers.
wait
```
I can provide the full original scripts.

**Anything else we need to know?**:

Not that I'm aware of, but I'm happy to try to provide additional requested information if requested.

**Environment**:

This is on Oak Ridge National Laboratory's Summit supercomputer.

- Dask version: 2023.6.1 - 2023.12.0
- Python version: 3.9.7
- Operating System: Red Hat Enterprise Linux release 8.2 (Ootpa)
- Install method (conda, pip, source): pip install into conda env

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.