dask / dask/dask-gateway

ci job failes on main due to upstream dependency changes

Open
#944 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

Heads up - the `local`, `pbs` and `slurm` jobs are failing on `main` right now, and on pretty much any PR that pulls fresh deps from PyPI. I ran into it on #941 and dug in a bit; it doesn't look like anything in gateway itself broke, just two dependencies that moved under us. The `k8s` and `kerberos` jobs are fine. (@consideRatio, this is the "other CI failure" you mentioned over there.)

The first one hits `local`. `distributed` 2026.6.0 removed the old `dask-scheduler` / `dask-worker` console scripts – they live under `dask scheduler` / `dask worker` now (dask/distributed#9240). Gateway still calls the old names, both as the `scheduler_cmd`/`worker_cmd` defaults in `backends/base.py` and hardcoded as `/opt/python/bin/dask-scheduler` in the pbs/slurm tests, so cluster startup just dies with:
```
FileNotFoundError: [Errno 2] No such file or directory: 'dask-scheduler'
```

The second one hits `pbs` and `slurm`, and it's actually why they don't even get to pytest. Pillow 12.3.0 stopped publishing `manylinux2014` wheels, and our CI image is still CentOS 7 (glibc 2.17), so pip grabs sdist and tries to build it – but there's no gcc in the container:
```
error: command 'gcc' failed: No such file or directory
```
Pillow only shows up here because `bokeh` drags it in.

The k8s and kerberos jobs dodge all of this: k8s builds from frozen images still pinned to `distributed==2026.3.0`, and the kerberos test never spins up a real cluster.

Fixing it isn't much: point the scheduler/worker commands at `dask scheduler` / `dask worker`, and add `pillow` to the `mamba install -c conda-forge ...` line the pbs/slurm `_install.sh` already uses for contourpy and pandas.

Full logs if useful: https://github.com/dask/dask-gateway/actions/runs/29776604255

I'm happy to send a PR for both – just say the word.

Contributor guide

Open the contributing guide

Research direction

Start with backends/base.py and the PBS/Slurm tests to trace the scheduler and worker commands, then inspect _install.sh for the existing mamba dependency list. Run the local, PBS, and Slurm CI jobs with fresh dependencies; done means they start clusters and pass without the missing-command or Pillow build failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.