JuliaParallel / JuliaParallel/SlurmClusterManager.jl

Slurm: -o argument in addprocs_slurm leads to an error

Open
#65 1 comment 0 reactions 0 assignees View on GitHub
manager: SLURM
Dominant language
Julia
Stars
76
Forks
10
Avg merge
8m
Merged PRs (30d)
1

Description

Hello everyone,

To add workers and schedule jobs on the cluster, I'm using the addprocs_slurm function from ClasterManagers

```
slurm_cpus = 4
@async addprocs(SlurmManager(slurm_cpus), partition="all", t="00:10:0")
```

It works as intended

```
Task (runnable) @0x00002b8be08c5cd0connecting to worker 1 out of 4

srun: job 13332841 queued and waiting for resources

julia> srun: job 13332841 has been allocated resources
connecting to worker 2 out of 4
connecting to worker 3 out of 4
connecting to worker 4 out of 4

```
However, if I have a lot of workers, the corresponding number of output files appears in the working directory. I decided to add the -o argument and log everything into one file

```
slurm_cpus = 4
@async addprocs(SlurmManager(slurm_cpus), partition="all", t="00:10:0", o="log.out")
```

It indeed creates this log file

```
julia_worker:9007#131.169.193.109
julia_worker:9006#131.169.193.109
julia_worker:9008#131.169.193.109
julia_worker:9009#131.169.193.109
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
```

but does not give any workers

```
Task (runnable) @0x00002b8be01f7260connecting to worker 1 out of 4

srun: job 13332876 queued and waiting for resources

julia> srun: job 13332876 has been allocated resources
srun: error: max-wn009: tasks 0-3: Exited with exit code 1
```

I decided to have a look at the source code. If I understand correctly, [it](https://github.com/JuliaParallel/ClusterManagers.jl/blob/master/src/slurm.jl) specifies values for -o and -D independently of my choice. Maybe, it causes trouble

```
jobname = "julia-$(getpid())"
job_output_name = "$(jobname)-$(trunc(Int, Base.time() * 10))"
make_job_output_path(task_num) = joinpath(job_file_loc, "$(job_output_name)-$(task_num).out")
job_output_template = make_job_output_path("%4t")
srun_cmd = `srun -J $jobname -n $np -o "$(job_output_template)" -D $exehome $(srunargs) $exename $exeflags $(worker_arg())`
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.