ci job failes on main due to upstream dependency changes
- Dominant language
- Python
- Stars
- 148
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Heads up - the `local`, `pbs` and `slurm` jobs are failing on `main` right now, and on pretty much any PR that pulls fresh deps from PyPI. I ran into it on #941 and dug in a bit; it doesn't look like anything in gateway itself broke, just two dependencies that moved under us. The `k8s` and `kerberos` jobs are fine. (@consideRatio, this is the "other CI failure" you mentioned over there.)
The first one hits `local`. `distributed` 2026.6.0 removed the old `dask-scheduler` / `dask-worker` console scripts – they live under `dask scheduler` / `dask worker` now (dask/distributed#9240). Gateway still calls the old names, both as the `scheduler_cmd`/`worker_cmd` defaults in `backends/base.py` and hardcoded as `/opt/python/bin/dask-scheduler` in the pbs/slurm tests, so cluster startup just dies with:
```
FileNotFoundError: [Errno 2] No such file or directory: 'dask-scheduler'
```
The second one hits `pbs` and `slurm`, and it's actually why they don't even get to pytest. Pillow 12.3.0 stopped publishing `manylinux2014` wheels, and our CI image is still CentOS 7 (glibc 2.17), so pip grabs sdist and tries to build it – but there's no gcc in the container:
```
error: command 'gcc' failed: No such file or directory
```
Pillow only shows up here because `bokeh` drags it in.
The k8s and kerberos jobs dodge all of this: k8s builds from frozen images still pinned to `distributed==2026.3.0`, and the kerberos test never spins up a real cluster.
Fixing it isn't much: point the scheduler/worker commands at `dask scheduler` / `dask worker`, and add `pillow` to the `mamba install -c conda-forge ...` line the pbs/slurm `_install.sh` already uses for contourpy and pandas.
Full logs if useful: https://github.com/dask/dask-gateway/actions/runs/29776604255
I'm happy to send a PR for both – just say the word.
Contributor guide
Research direction
Start with backends/base.py and the PBS/Slurm tests to trace the scheduler and worker commands, then inspect _install.sh for the existing mamba dependency list. Run the local, PBS, and Slurm CI jobs with fresh dependencies; done means they start clusters and pass without the missing-command or Pillow build failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, ci-cd
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100