SpikeInterface / SpikeInterface/spikeinterface
Stray .nfs file preventing deletion of waveforms extension folder
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
Hi all — I'm on a Linux / NFS system (CentOS 7 specifically, on my school's HPCC), and I encounter an OSError every time I try to re-compute the waveforms extension of a sorting analyzer. The core problem is that there's a stray .nfs temporary file hanging out in the waveforms folder. If I do ls -a [that folder] I can see it clearly, it's always something like .nfs9eb83351caf602ec00005734.
I can see that it's python using the file with lsof:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 30135 jop9552 mem REG 0,45 128896688 11436947680097862380 [...]/analyzer_folder/extensions/waveforms/.nfs9eb83351caf602ec00005734
python 30135 jop9552 93u REG 0,45 128896688 11436947680097862380 [...]/analyzer_folder/extensions/waveforms/.nfs9eb83351caf602ec00005734
and if I call kill on that process, my notebook's kernel crashes, so I'm pretty sure it's coming from spikeinterface.
The offending function is at /spikeinterface/core/sortinganalyzer.py:2090: shutil.rmtree(extension_folder).
Traceback
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
Cell In[6], line 1
----> 1 analyzer.compute("waveforms")
File ~/datta-lab/spikeinterface/src/spikeinterface/core/sortinganalyzer.py:1155, in SortingAnalyzer.compute(self, input, save, extension_params, verbose, **kwargs)
1108 """
1109 Compute one extension or several extensiosn.
1110 Internally calls compute_one_extension() or compute_several_extensions() depending on the input type.
(...)
1152
1153 """
1154 if isinstance(input, str):
-> 1155 return self.compute_one_extension(extension_name=input, save=save, verbose=verbose, **kwargs)
1156 elif isinstance(input, dict):
1157 params_, job_kwargs = split_job_kwargs(kwargs)
File ~/datta-lab/spikeinterface/src/spikeinterface/core/sortinganalyzer.py:1232, in SortingAnalyzer.compute_one_extension(self, extension_name, save, verbose, **kwargs)
1229 assert ok, f"Extension {extension_name} requires {dependency_name} to be computed first"
1231 extension_instance = extension_class(self)
-> 1232 extension_instance.set_params(save=save, **params)
1233 if extension_class.need_job_kwargs:
1234 extension_instance.run(save=save, verbose=verbose, **job_kwargs)
File ~/datta-lab/spikeinterface/src/spikeinterface/core/sortinganalyzer.py:2053, in AnalyzerExtension.set_params(self, save, **params)
2047 """
2048 Set parameters for the extension and
2049 make it persistent in json.
2050 """
2051 # this ensure data is also deleted and corresponf to params
2052 # this also ensure the group is created
-> 2053 self._reset_extension_folder()
2055 params = self._set_params(**params)
2056 self.params = params
File ~/datta-lab/spikeinterface/src/spikeinterface/core/sortinganalyzer.py:2028, in AnalyzerExtension._reset_extension_folder(self)
2026 extension_folder = self._get_binary_extension_folder()
2027 if extension_folder.is_dir():
-> 2028 shutil.rmtree(extension_folder)
2029 extension_folder.mkdir(exist_ok=False, parents=True)
2031 elif self.format == "zarr":
File ~/miniconda3/envs/spikeinterface/lib/python3.9/shutil.py:740, in rmtree(path, ignore_errors, onerror)
738 os.rmdir(path)
739 except OSError:
--> 740 onerror(os.rmdir, path, sys.exc_info())
741 else:
742 try:
743 # symlinks to directories are forbidden, see bug #1669
File ~/miniconda3/envs/spikeinterface/lib/python3.9/shutil.py:738, in rmtree(path, ignore_errors, onerror)
736 os.close(fd)
737 fd_closed = True
--> 738 os.rmdir(path)
739 except OSError:
740 onerror(os.rmdir, path, sys.exc_info())
OSError: [Errno 39] Directory not empty: '[...]/analyzer_folder/extensions/waveforms'
MRE:
recording = si.load_extractor([any recording])
sorting = si.read_sorter_folder([any sorting])
analyzer = si.create_sorting_analyzer(
sorting=sorting,
recording=recording,
format='binary_folder',
folder="./tmp_analyzer",
)
to_compute = ["random_spikes", "waveforms"]
analyzer.compute(to_compute)
analyzer.compute("waveforms") # tries to reset folder before recomputing but fails at deletion
Not sure how to go about debugging this but any advice is welcome. This is v0.101.0. Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/spikeinterface/core/sortinganalyzer.py at AnalyzerExtension._reset_extension_folder() and the shutil.rmtree call around line 2028. Reproduce the MRE on a Linux/NFS system, then investigate the .nfs file behavior during waveforms recomputation. Done means recomputing the waveforms extension no longer fails with Directory not empty in this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100