Error in Singularity container: numcodecs.blosc import failure

Description:

I am encountering an issue when running spike sorting on large datasets (278GB) with Kilosort4 in a Singularity container. The same pipeline works perfectly with smaller datasets (2GB), but fails with this larger one.

Environment:

• Platform: MIT OpenMind computing cluster
• Python version: 3.11
• SpikeInterface version: 0.102.1
• Singularity version: 3.9.5

Issue:

The sorting step fails with the following error:

/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py:55: DeprecationWarning: load_extractor() is deprecated and will be removed in the future. Please use load() instead.
preprocessed_recordings[group] = si.load_extractor(recording_path)
Traceback (most recent call last):
File "/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py", line 120, in <module>
main(args)
File "/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py", line 102, in main
sorting = si.run_sorter(
^^^^^^^^^^^^^^
File "/home/jchen13/.conda/envs/si_new/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 193, in run_sorter
return run_sorter_container(
^^^^^^^^^^^^^^^^^^^^^
File "/home/jchen13/.conda/envs/si_new/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 648, in run_sorter_container
raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_sorter_script.py", line 4, in <module>
from spikeinterface import load
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/**init**.py", line 7, in <module>
from .core import *
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/**init**.py", line 20, in <module>
from .zarrextractors import ZarrRecordingExtractor, ZarrSortingExtractor, read_zarr, get_default_zarr_compressor
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/zarrextractors.py", line 6, in <module>
import zarr
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/**init**.py", line 3, in <module>
from zarr.convenience import (
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/convenience.py", line 8, in <module>
from zarr._storage.store import data_root, meta_root, assert_zarr_v3_api_available
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/_storage/store.py", line 10, in <module>
from zarr.meta import Metadata2, Metadata3
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/meta.py", line 10, in <module>
from zarr.util import json_dumps, json_loads
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/util.py", line 34, in <module>
from numcodecs.blosc import cbuffer_sizes, cbuffer_metainfo
ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/sclib/python3.11/site-packages/numcodecs/blosc.cpython-311-x86_64-linux-gnu.so)

This is how I am calling the sorter with Singularity in my sorting.py file which is attached to this issue:

 # Run the sorter
        sorting = si.run_sorter(
            sorter_name='spikeinterface/kilosort4-base:4.0.18_cuda-12.0.0',
            recording=sub_recording,
            folder=ks_sub_folder,
            singularity_image=True, # Update as necessary
            verbose=True,
            remove_existing_folder=True,
            **params
            )

sorting.py.pdf

Any guidance on how to resolve this issue would be greatly appreciated. We're happy to provide additional information if needed.

Start with sorting.py around lines 102 and 120, then trace the Singularity execution through in_container_sorter_script.py and the zarr/util.py import shown in the traceback. Reproduce the failure with Python 3.11, SpikeInterface 0.102.1, and the stated container path; done means the Kilosort4 Singularity run imports spikeinterface successfully and no longer fails at numcodecs.blosc.