Parent job in slurm exits before child jobs | nested slurm jobs

I was trying to setup a small test project to use batchtools on slurm. I am having an issue that the parent job exits from slurm before all the child jobs are completed. How can I solve this issue?

The main Rscript that submits jobs and the associated configuration files are as:

run_batchtools_job.R

library(batchtools)

reg <- makeRegistry(file.dir =  "slurm_registry", seed = 5081, conf.file = "Scripts/batch_tools_test/.batchtools.conf.R")

my_fun <- function(x) {
  Sys.sleep(x)  
  return(x^2)
}

ids <- batchMap(fun = my_fun, x = 100:150, reg = reg)
done <- submitJobs(ids = ids, reg = reg, resources = list(partition = "small", walltime = 86400, memory = 1024, ntasks = 1))
waitForJobs(ids = ids, reg = reg) 
getStatus(ids = ids, reg = reg)    

final_res <- reduceResultsList(ids = ids, reg = reg)
print(class(final_res))

.batchtools.conf.R

cluster.functions <- makeClusterFunctionsSlurm(template = "Scripts/batch_tools_test/slurm_config.tmpl", 
                                               array.jobs = TRUE, 
                                               scheduler.latency = 60,
                                               fs.latency = 30)
max.concurrent.jobs <- 5

slurm_config.tmpl

#!/bin/bash
#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --ntasks=<%= resources$ntasks %>
#SBATCH --mem=<%= resources$memory %>MB
#SBATCH --partition=<%= resources$partition %>

module load  r/4.3.3

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

I submit the run_batchtools_job.R script to slurm using the following sbatch script.

run_batchtools.sh

#!/bin/bash
#SBATCH --job-name=batchtools_test
#SBATCH --output=batchtools_test.log
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --mem=2G
#SBATCH --partition=small

# Load R
module load  r/4.3.3

# Run your R script
Rscript Scripts/batch_tools_test/run_batchtools_job.R

I observed that the batchtools_test job exits before all the child jobs spawned using submitJobs end. As a result, there is nothing in final_res.

While checking getErrorMessages, I saw that several jobs are listed as 'not terminated'. But when I manually checked the logs and the results within the registry directories, everything completed as expected.

How can I overcome this issue?

Start with run_batchtools_job.R and the batchtools configuration in .batchtools.conf.R, then inspect the Slurm template and submitting script for the parent and child job lifecycles. Read the behavior around waitForJobs and getErrorMessages, including scheduler and filesystem latency settings. Done means the parent waits for all child jobs, reports them as terminated, and populates final_res.