dask / dask/dask-jobqueue

CI: Distributed fixtures not compliant with dask-jobqueue

Open
#569 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

I hesitated to put this issue as a comment in #567, but I'm under the impression we have two separate problems.

In #562, I override the `cleanup` fixture from distributed to prevent it from removing the default MainThread loop (so #567 related), but also to make it reload the dask-jobqueue configuration into `dask.config`. Since distributed last 2022.08.0 version, the loop fixture we used into our tests has also been modified (https://github.com/dask/distributed/commit/172e37f41f84f117ec79a1186a951be6f66db7f3). In this new version, config is also reset, so I directly modify the method `distributed.utils_test.reset_config` to also reload dask-jobqueue config.

Question is: how could we use these fixtures without these hacks? How to make dask-jobqueue config reset at the same time as the distributed one is reset? Does this need a fix in distributed?

Going further, I have another question of understanding (also expressed [here](https://github.com/dask/dask-jobqueue/pull/562#issuecomment-1207342825)): why do we need the `loop` fixture? I don't see other Dask cluster deploying system using it. Is this still necessary with SpecCluster?

cc @graingert @jacobtomlinson.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.