SpikeInterface / SpikeInterface/spikeinterface

Pick better default for mp_context

Open
#4,422 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

io performance
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

#4333 changes how mp_context is chosen from:

https://github.com/SpikeInterface/spikeinterface/blob/d8f49ece6bc94cf84beb15840e07250acbbcb80d/src/spikeinterface/core/job_tools.py#L419-L427

to

https://github.com/SpikeInterface/spikeinterface/blob/81e89fd19cffdbf031e48c54100b6ab9e024e3ca/src/spikeinterface/core/job_tools.py#L418-L440

Before the PR; job_kwargs.mp_context=None and recording.get_preferred_mp_context()=None (the default and most common setup) will call mp_context = multiprocessing.get_context(None) that returned the default context for that specific machine (spawn for windows & Mac and fork/forkserver for Linux). Now, the behavior has changed for Linux: it always picks fork. Default mp_context for Linux from python 3.14 onwards is forkserver (a thread-safe, less problematic alternative to fork) docs.

When running kilosort4 in python 3.14. fork did not play nice with some multi-threaded calls inside scikit learn, particularly the process hangs forever in this line:
https://github.com/MouseLand/Kilosort/blob/7a19a57bef39f3e07ab7fa1edf5aa8635c69a850/kilosort/spikedetect.py#L72
This was run inside a kubernetes pod with GPUs but I couldn't replicate it locally: 3.13 and 3.14 both worked with fork or forkserver :/ so not sure if it will be a problem for other people.

In any case, it might be worth considering making forkserver (rather than fork) the Linux default (it's been around since python 3.4) or allow multiprocessing to pick the best default based on system and python version as it did before this PR, i.e., let mp_context=None with a comment saying what that means).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/spikeinterface/core/job_tools.py around lines 418-440 and compare the earlier selection at lines 419-427. Review Python's multiprocessing context defaults and the reported Kilosort/scikit-learn hang, especially the referenced spikedetect.py line 72. Done means agreeing on and implementing a safe default or preserving None with clear documentation, then validating the affected workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.