mne-tools / mne-tools/mne-python
[enh] read_raw_brainvision can load nonsense data - add proper error or warning?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 1.6k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 100
Description
Describe the bug
hi all -
this is not a serious bug, and maybe you are all aware of it, but i wanted to raise it anyway:
the read_raw_brainvision function / RawBrainVision class can read in corrupted or nonsense 'binary' files without raising an error.
see also this thread in the mne-bids repo
probably the current code simply assumes the provided file is correct, opens it as binary, and interprets the content accordingly.
if there were an easy way to catch such cases, we could give more informative errors or warnings than there are right now (see below).
i think it should be possible to quickly validate the data when loading it, as the header file holds the relevant information for this (i.e. the number of DataPoints).
this is not high priority, though, as we should generally trust the users to supply meaningful data..
(relatedly, there should also not be any security risks via malevolent users, as i suppose the content of the binary files is never executed?)
Steps to reproduce
unpack this example data - the binary file containing EEG data is replaced by a random renamed json file:
Archive.zip
from mne.io import read_raw_brainvision
raw = read_raw_brainvision('path/to/sub-001_task-xxxx_eeg.vhdr', preload=True)
Expected results
an informative error message or warning would be great.
in this specific case there is already a warning that annotations are omitted, because there is not enough data (see below), but this would fail if there are few/no annotations, or if the corrupted file is large enough (i guess)
Actual results
the call runs without an error.
a warning is raised that there are annotations not matched by data:
Out: RuntimeWarning: Omitted 1200 annotation(s) that were outside data range.
..but the function loads 'data' with one sample per channel.
raw.get_data().shape
Out: (72, 1)
in case of an empty or very short textfile provided as 'binary' .eeg (not included in the example data) a generic error is raised:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 r = mne.io.read_raw_brainvision('/Users/dominik.welke/Work/11_datasets/mne-bids-brainvision-import/sub-001_task-xxxx_eeg.vhdr', preload=True)
File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/brainvision/brainvision.py:859, in read_raw_brainvision(vhdr_fname, eog, misc, scale, preload, verbose)
825 @fill_doc
826 def read_raw_brainvision(vhdr_fname,
827 eog=('HEOGL', 'HEOGR', 'VEOGb'), misc='auto',
828 scale=1., preload=False, verbose=None):
829 """Reader for Brain Vision EEG file.
830
831 Parameters
(...)
857 mne.io.Raw : Documentation of attribute and methods.
858 """
--> 859 return RawBrainVision(vhdr_fname=vhdr_fname, eog=eog,
860 misc=misc, scale=scale, preload=preload,
861 verbose=verbose)
File <decorator-gen-220>:12, in __init__(self, vhdr_fname, eog, misc, scale, preload, verbose)
File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/brainvision/brainvision.py:95, in RawBrainVision.__init__(self, vhdr_fname, eog, misc, scale, preload, verbose)
91 n_samples = n_samples // (dtype_bytes * n_data_ch)
93 raw_extras = dict(
94 offsets=offsets, fmt=fmt, order=order, n_samples=n_samples)
---> 95 super(RawBrainVision, self).__init__(
96 info, last_samps=[n_samples - 1], filenames=[data_fname],
97 orig_format=fmt, preload=preload, verbose=verbose,
98 raw_extras=[raw_extras], orig_units=orig_units)
100 self.set_montage(montage)
102 settings, cfg, cinfo, _ = _aux_vhdr_info(vhdr_fname)
File <decorator-gen-203>:12, in __init__(self, info, preload, first_samps, last_samps, filenames, raw_extras, orig_format, dtype, buffer_size_sec, orig_units, verbose)
File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/base.py:287, in BaseRaw.__init__(self, info, preload, first_samps, last_samps, filenames, raw_extras, orig_format, dtype, buffer_size_sec, orig_units, verbose)
285 # If we have True or a string, actually do the preloading
286 if load_from_disk:
--> 287 self._preload_data(preload)
288 self._init_kwargs = _get_argvalues()
File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/base.py:566, in BaseRaw._preload_data(self, preload)
563 if isinstance(preload, (bool, np.bool_)) and not preload:
564 data_buffer = None
565 logger.info('Reading %d ... %d = %9.3f ... %9.3f secs...' %
--> 566 (0, len(self.times) - 1, 0., self.times[-1]))
567 self._data = self._read_segment(
568 data_buffer=data_buffer, projector=self._projector)
569 assert len(self._data) == self.info['nchan']
IndexError: index -1 is out of bounds for axis 0 with size 0
Additional information
sys_info:
Platform: macOS-10.16-x86_64-i386-64bit
Python: 3.9.12 (main, Apr 5 2022, 01:53:17) [Clang 12.0.0 ]
Executable: /opt/anaconda3/envs/mne-bids-dev/bin/python
CPU: i386: 4 cores
Memory: Unavailable (requires "psutil" package)
mne: 1.0.3
numpy: 1.22.3 {}
scipy: 1.8.0
matplotlib: 3.5.2 {backend=MacOSX}
sklearn: Not found
numba: Not found
nibabel: Not found
nilearn: Not found
dipy: Not found
cupy: Not found
pandas: 1.4.2
pyvista: Not found
pyvistaqt: Not found
ipyvtklink: Not found
vtk: Not found
PyQt5: Not found
ipympl: Not found
pooch: v1.6.0
mne_bids: 0.11.dev0
mne_nirs: Not found
mne_features: Not found
mne_qt_browser: Not found
mne_connectivity: Not found
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the read_raw_brainvision function and RawBrainVision class, then reproduce the issue using the linked Archive.zip and the provided vhdr/eeg example. Check how the header's DataPoints value relates to the loaded binary data. Done means corrupted or insufficient binary content produces an informative error or warning instead of silently loading nonsense data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100