dask / dask/dask-jobqueue

Potential for SLURMCluster log directory creation failing

Open
#692 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

I am using `dask_jobqueue.SLURMCluster` in my workflows. I am running ~15 workflows at a time. I am getting unlucky and it seems like two separate workflows are both trying to create the output log directory. One would fail:

```
File "/datasets/work/jones-storage/work/miniconda/miniforge3/envs/flint_main/lib/python3.12/site-packages/dask_jobqueue/core.py", line 661, in __init__
self._dummy_job # trigger property to ensure that the job is valid
^^^^^^^^^^^^^^^
File "/datasets/work/jones-storage/work/miniconda/miniforge3/envs/flint_main/lib/python3.12/site-packages/dask_jobqueue/core.py", line 690, in _dummy_job
return self.job_cls(
^^^^^^^^^^^^^
File "/datasets/work/jones-storage/work/miniconda/miniforge3/envs/flint_main/lib/python3.12/site-packages/dask_jobqueue/slurm.py", line 37, in __init__
super().__init__(
File "/datasets/work/jones-storage/work/miniconda/miniforge3/envs/flint_main/lib/python3.12/site-packages/dask_jobqueue/core.py", line 375, in __init__
os.makedirs(self.log_directory)
File "", line 225, in makedirs
FileExistsError: [Errno 17] File exists: 'flint_logs'
```

**Minimal Complete Verifiable Example**:

Given the 'race condition' involved I am not able to give a concise single example. I am guessing that it is a case that the two workflows are both getting past:

https://github.com/dask/dask-jobqueue/blob/d562e6c8c0560f8ab84d12edb70726f536fd551f/dask_jobqueue/core.py#L379

and subsequently only one is successfully able to make the directory.

Would it make sense to use `Pathlib.Path` here? Something like

```
if self.log_directory:
log_path = Path(self.log_directory)
log_path.mkdir(exists_ok=True, parent=True)
```

**Anything else we need to know?**:

**Environment**:

- Dask version:
- Python version: Python 3.12.8
- Operating System: SLES 15.5
- Install method (conda, pip, source): pip
- dask_jobqueue: 0.9.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.