JuliaParallel / JuliaParallel/SlurmClusterManager.jl
Slurm: -o argument in addprocs_slurm leads to an error
- Dominant language
- Julia
- Stars
- 76
- Forks
- 10
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
Hello everyone,
To add workers and schedule jobs on the cluster, I'm using the addprocs_slurm function from ClasterManagers
```
slurm_cpus = 4
@async addprocs(SlurmManager(slurm_cpus), partition="all", t="00:10:0")
```
It works as intended
```
Task (runnable) @0x00002b8be08c5cd0connecting to worker 1 out of 4
srun: job 13332841 queued and waiting for resources
julia> srun: job 13332841 has been allocated resources
connecting to worker 2 out of 4
connecting to worker 3 out of 4
connecting to worker 4 out of 4
```
However, if I have a lot of workers, the corresponding number of output files appears in the working directory. I decided to add the -o argument and log everything into one file
```
slurm_cpus = 4
@async addprocs(SlurmManager(slurm_cpus), partition="all", t="00:10:0", o="log.out")
```
It indeed creates this log file
```
julia_worker:9007#131.169.193.109
julia_worker:9006#131.169.193.109
julia_worker:9008#131.169.193.109
julia_worker:9009#131.169.193.109
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
```
but does not give any workers
```
Task (runnable) @0x00002b8be01f7260connecting to worker 1 out of 4
srun: job 13332876 queued and waiting for resources
julia> srun: job 13332876 has been allocated resources
srun: error: max-wn009: tasks 0-3: Exited with exit code 1
```
I decided to have a look at the source code. If I understand correctly, [it](https://github.com/JuliaParallel/ClusterManagers.jl/blob/master/src/slurm.jl) specifies values for -o and -D independently of my choice. Maybe, it causes trouble
```
jobname = "julia-$(getpid())"
job_output_name = "$(jobname)-$(trunc(Int, Base.time() * 10))"
make_job_output_path(task_num) = joinpath(job_file_loc, "$(job_output_name)-$(task_num).out")
job_output_template = make_job_output_path("%4t")
srun_cmd = `srun -J $jobname -n $np -o "$(job_output_template)" -D $exehome $(srunargs) $exename $exeflags $(worker_arg())`
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.