mne-tools / mne-tools/mne-python

BrokenProcessPool with compute_covariance on loaded epochs

Open
#12,028 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

BUG
Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

Description of the problem

The cross-validation step to select the best estimator fails in joblib. Setting n_jobs=1 does by-pass the problem, but it would still be interesting to figure out what is going wrong here.

Traceback 
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/scheltie/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/externals/loky/process_executor.py", line 426, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/home/scheltie/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/numpy_pickle.py", line 600, in load_temporary_memmap
    add_maybe_unlink_finalizer(obj)
  File "/home/scheltie/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/_memmapping_reducer.py", line 72, in add_maybe_unlink_finalizer
    "".format(type(memmap), id(memmap), os.path.basename(memmap.filename),
  File "/usr/lib/python3.10/posixpath.py", line 142, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
"""


The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  Cell In[2], line 5
    cov = compute_covariance(

  File <decorator-gen-219>:12 in compute_covariance

  File ~/git/mscheltienne/mne-python/mne/cov.py:1161 in compute_covariance
    cov_data = _compute_covariance_auto(

  File ~/git/mscheltienne/mne-python/mne/cov.py:1386 in _compute_covariance_auto
    loglik = _cross_val(data, estimator, cv, n_jobs)

  File ~/git/mscheltienne/mne-python/mne/cov.py:1419 in _cross_val
    cross_val_score(

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:562 in cross_val_score
    cv_results = cross_validate(

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/sklearn/utils/_param_validation.py:211 in wrapper
    return func(*args, **kwargs)

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:309 in cross_validate
    results = parallel(

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/sklearn/utils/parallel.py:65 in __call__
    return super().__call__(iterable_with_config)

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:1952 in __call__
    return output if self.return_generator else list(output)

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:1595 in _get_outputs
    yield from self._retrieve()

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:1699 in _retrieve
    self._raise_error_fast()

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:1734 in _raise_error_fast
    error_job.get_result(self.timeout)

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:736 in get_result
    return self._return_or_raise()

  File ~/pyvenv/mscheltienne/mne-python/lib/python3.10/site-packages/joblib/parallel.py:754 in _return_or_raise
    raise self._result

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Steps to reproduce
from mne import compute_covariance, read_epochs


epochs = read_epochs("/home/scheltie/Downloads/short-epo.fif")
cov = compute_covariance(
    epochs,
    tmin=None,
    tmax=0,
    method="auto",  # ('shrunk', 'diagonal_fixed', 'empirical', 'factor_analysis')
    cv=3,
    n_jobs=6,
)

Interestingly, I can not reproduce with the sample dataset..

Link to data

short-epo.zip

Additional information

Fresh environment with MNE-main.

mne.sys_info() 
Platform             Linux-6.4.6-76060406-generic-x86_64-with-glibc2.35
Python               3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Executable           /home/scheltie/pyvenv/mscheltienne/mne-python/bin/python
CPU                  x86_64 (12 cores)
Memory               31.0 GB

Core
├☑ mne               1.6.0.dev126+gf3bb56d1d
├☑ numpy             1.26.0 (OpenBLAS 0.3.23.dev with 12 threads)
├☑ scipy             1.11.3
├☑ matplotlib        3.8.0 (backend=Qt5Agg)
├☑ pooch             1.7.0
└☑ jinja2            3.1.2

Numerical (optional)
├☑ sklearn           1.3.1
├☑ nibabel           5.1.0
├☑ nilearn           0.10.1
├☑ dipy              1.7.0
├☑ pandas            2.1.1
└☐ unavailable       numba, openmeeg, cupy

Visualization (optional)
├☑ pyvista           0.42.2 (OpenGL 4.6 (Core Profile) Mesa 23.1.3-1pop0~1689084530~22.04~0618746 via Mesa Intel(R) UHD Graphics 770 (ADL-S GT1))
├☑ pyvistaqt         0.11.0
├☑ vtk               9.2.6
├☑ qtpy              2.4.0 (PyQt5=5.15.2)
└☐ unavailable       ipympl, pyqtgraph, mne-qt-browser, ipywidgets, trame_client, trame_server, trame_vtk, trame_vuetify

Ecosystem (optional)
├☑ mne-bids          0.14.dev0
├☑ mne-connectivity  0.6.0dev0
└☐ unavailable       mne-nirs, mne-features, mne-icalabel, mne-bids-pipeline

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mne/cov.py at _compute_covariance_auto and _cross_val, then reproduce with the attached short-epo.zip using compute_covariance with method="auto", cv=3, and n_jobs=6. Compare the failing parallel run with n_jobs=1 and the sample dataset; done means loaded epochs complete cross-validation without BrokenProcessPool.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.