Probability function returned NaN

Good morning,

I have an issue on emcee. The EnsembleSampler crashes after a few iterations and raises a ValueError: Probability function returned NaN. The thing is my user-defined probability function cannot return NaNs, as it defined something like this:

def my_probability_function(x):
    prob = logP(*x)
    if prob > 0 or np.isnan(prob): return -np.inf
    return prob

where logP is basically an evaluation of a beforehand-constructed RBF interpolator.

I call the EnsembleSampler like this:

import emcee
import numpy as np
from pathos.pools import ProcessPool

NW = 64 #Number of workers
ND = 7 #Number of dimensions
MCMC_PATH = 'path/to/backend.h5'
backend = emcee.backends.HDFBackend(MCMC_PATH)
backend.reset(NW, ND)
with ProcessPool(nodes=8) as pool:
    pool.restart()
    sampler = emcee.EnsembleSampler(NW, ND, my_probability_function, pool=pool, backend=backend)
    sampler.run_mcmc(p0, 5000, progress=True)

Changing the logP function to e.g. a nearest interpolator solves the problem, so I guess the issue comes from a bad interaction between emcee and scipy's RBF interpolator?

I tried advancing the MCMC iteration by iteration and comparing at each step the manual and emcee-stored evaluation of the probability function:

sampler.run_mcmc(pp, 1, skip_initial_state_check=True)
pp = sampler.get_chain(flat=True)[-NW:]

logP_manual = [my_probability_function(p) for p in pp]
logP_emcee, _ = sampler.compute_log_prob(pp)

diff = np.abs(1. - logP_manual/logP_emcee) 

and at some point, diff was not identically zero, meaning that my_probability_function sometimes returns different results depending on if one evaluates it outside or inside emcee!

Can you help me please? Unfortunately, I was not able to replicate the problem in a minimalist example, as it only emerges when the RBF interpolator is constructed on my personal 2M data points in 7 dimensions.

Many thanks in advance.

Start with EnsembleSampler.run_mcmc and compute_log_prob, comparing direct calls to the RBF interpolator with values stored by emcee. The investigation needs a minimal reproducer or a controlled comparison using the reported 2M-point, 7-dimensional setup; done means identifying whether the discrepancy is in emcee or the scipy RBF interaction.