`mindeps` CI environment includes `numpy` and `pandas`

From what I understand, the `mindeps` CI environment is not supposed to include `numpy` or `pandas`, that's what `mindeps-numpy` and `mindeps-pandas` are for, respectively. They still get installed though:

`numpy`: https://github.com/dask/distributed/actions/runs/8982581212/job/24670508020#step:12:90
`pandas`: https://github.com/dask/distributed/actions/runs/8982581212/job/24670508020#step:12:94

We should figure out why they get installed in the first place and remove them to make sure that our CI covers what we're interested in.