Parallelization of emcee is working not as fast as expected

General information:

• emcee version: 3.1.3
• platform: JupyterHub, running Linux
• installation method: pip

Problem description:

Expected behavior:

Hi all,

I was following this example for Parallelization using Multiprocessing. From the documentation, the code should run 3.3 times faster in parallel, compared to running in serial, given that we have a 4-core CPU.

Actual behavior:

This didn't happen on my side. When using 8 cores, I only obtained a 1.1x performance boost.

Minimal example:

import emcee
import os
os.environ["OMP_NUM_THREADS"] = "1"
import time
import numpy as np

def log_prob(theta):
    t = time.time() + np.random.uniform(0.005, 0.008)
    while True:
        if time.time() >= t:
            break
    return -0.5 * np.sum(theta**2)

np.random.seed(42)

initial = np.random.randn(32, 5)
nwalkers, ndim = initial.shape
nsteps = 100

sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob)
start = time.time()
sampler.run_mcmc(initial, nsteps, progress=True)
end = time.time()
serial_time = end - start
print("Serial took {0:.1f} seconds".format(serial_time))

from multiprocessing import Pool

with Pool() as pool:
    sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, pool=pool)
    start = time.time()
    sampler.run_mcmc(initial, nsteps, progress=True)
    end = time.time()
    multi_time = end - start
    print("Multiprocessing took {0:.1f} seconds".format(multi_time))
    print("{0:.1f} times faster than serial".format(serial_time / multi_time))

Please see the attachment for the notebook I was using.
emceeTest.zip

Start by running the minimal Python example in the issue and comparing its serial and multiprocessing timings on Linux. Read the linked parallelization documentation and inspect the attached emceeTest notebook for differences in setup. Done means identifying why the observed speedup differs from the documented expectation and documenting or correcting the behavior.