MPIPool: Child processes continue when parent job is deleted on cluster

General information:

• emcee version: 2.2.1
• platform: Linux
• installation method (pip/conda/source/other?): pip

Problem description:

I am running a 30-core job on a cluster using emcee. When I delete the job through the ubiquitous qdel <job ID> command, only the parent is killed, while the child processes continue to run. In other words: qdel makes the job-ID disappear from the queue, but the 30 (= number of cores) initiated Python processes remain present in the background, contributing heavily to the cluster load. Furthermore, I can only ''manually'' kill the background processes on the one node I am logged into.

What have you tried so far?:

I have consulted the emcee documentation and the github source code relating to MPIPool, which I use to parallelise the job, but I did not manage to pinpoint where exactly things go wrong. Ideally, I would like a child process to call on its parent regularly and to terminate itself when it notices that the parent is no longer there. At the moment it seems that deleting the job simply cuts the rope that ties parent and child together, without deleting the child.

If you know what should be changed, I would be delighted to hear it!

Minimal example:

The snippet below shows how the pool object is integrated my main script. In addition I use a bash script to submit a job to the cluster queue (containing echo 'mpirun -np '$NCORES' python '$SKRIPTNAME >> $TMPFILE) and request the number of cores I want to use. The latter should work fine.

import emcee
from emcee.utils import MPIPool

pool = MPIPool()

if not pool.is_master():
    pool.wait()
    sys.exit(0)

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, pool = pool)
pos, prob, state = sampler.run_mcmc(p0, 1000) # p0 contains the initial walker positions

pool.close()

Start at the MPIPool entry point used in the example, particularly its master/worker lifecycle and the pool.wait() path. Reproduce the qdel behavior on a cluster if possible and trace what happens to workers when the parent disappears. Done means deleting the queued job also terminates the child Python processes without manual intervention.