mne-tools / mne-tools/mne-python

[enh] read_raw_brainvision can load nonsense data - add proper error or warning?

Open
#10,691 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

BUG
Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

Describe the bug

hi all -

this is not a serious bug, and maybe you are all aware of it, but i wanted to raise it anyway:

the read_raw_brainvision function / RawBrainVision class can read in corrupted or nonsense 'binary' files without raising an error.

see also this thread in the mne-bids repo

probably the current code simply assumes the provided file is correct, opens it as binary, and interprets the content accordingly.
if there were an easy way to catch such cases, we could give more informative errors or warnings than there are right now (see below).

i think it should be possible to quickly validate the data when loading it, as the header file holds the relevant information for this (i.e. the number of DataPoints).

this is not high priority, though, as we should generally trust the users to supply meaningful data..
(relatedly, there should also not be any security risks via malevolent users, as i suppose the content of the binary files is never executed?)

Steps to reproduce

unpack this example data - the binary file containing EEG data is replaced by a random renamed json file:
Archive.zip

from mne.io import read_raw_brainvision

raw = read_raw_brainvision('path/to/sub-001_task-xxxx_eeg.vhdr', preload=True)
Expected results

an informative error message or warning would be great.

in this specific case there is already a warning that annotations are omitted, because there is not enough data (see below), but this would fail if there are few/no annotations, or if the corrupted file is large enough (i guess)

Actual results

the call runs without an error.

a warning is raised that there are annotations not matched by data:

Out: RuntimeWarning: Omitted 1200 annotation(s) that were outside data range.

..but the function loads 'data' with one sample per channel.

raw.get_data().shape
Out: (72, 1)

in case of an empty or very short textfile provided as 'binary' .eeg (not included in the example data) a generic error is raised:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 r = mne.io.read_raw_brainvision('/Users/dominik.welke/Work/11_datasets/mne-bids-brainvision-import/sub-001_task-xxxx_eeg.vhdr', preload=True)

File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/brainvision/brainvision.py:859, in read_raw_brainvision(vhdr_fname, eog, misc, scale, preload, verbose)
    825 @fill_doc
    826 def read_raw_brainvision(vhdr_fname,
    827                          eog=('HEOGL', 'HEOGR', 'VEOGb'), misc='auto',
    828                          scale=1., preload=False, verbose=None):
    829     """Reader for Brain Vision EEG file.
    830 
    831     Parameters
   (...)
    857     mne.io.Raw : Documentation of attribute and methods.
    858     """
--> 859     return RawBrainVision(vhdr_fname=vhdr_fname, eog=eog,
    860                           misc=misc, scale=scale, preload=preload,
    861                           verbose=verbose)

File <decorator-gen-220>:12, in __init__(self, vhdr_fname, eog, misc, scale, preload, verbose)

File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/brainvision/brainvision.py:95, in RawBrainVision.__init__(self, vhdr_fname, eog, misc, scale, preload, verbose)
     91         n_samples = n_samples // (dtype_bytes * n_data_ch)
     93 raw_extras = dict(
     94     offsets=offsets, fmt=fmt, order=order, n_samples=n_samples)
---> 95 super(RawBrainVision, self).__init__(
     96     info, last_samps=[n_samples - 1], filenames=[data_fname],
     97     orig_format=fmt, preload=preload, verbose=verbose,
     98     raw_extras=[raw_extras], orig_units=orig_units)
    100 self.set_montage(montage)
    102 settings, cfg, cinfo, _ = _aux_vhdr_info(vhdr_fname)

File <decorator-gen-203>:12, in __init__(self, info, preload, first_samps, last_samps, filenames, raw_extras, orig_format, dtype, buffer_size_sec, orig_units, verbose)

File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/base.py:287, in BaseRaw.__init__(self, info, preload, first_samps, last_samps, filenames, raw_extras, orig_format, dtype, buffer_size_sec, orig_units, verbose)
    285 # If we have True or a string, actually do the preloading
    286 if load_from_disk:
--> 287     self._preload_data(preload)
    288 self._init_kwargs = _get_argvalues()

File /opt/anaconda3/envs/mne-bids-dev/lib/python3.9/site-packages/mne/io/base.py:566, in BaseRaw._preload_data(self, preload)
    563 if isinstance(preload, (bool, np.bool_)) and not preload:
    564     data_buffer = None
    565 logger.info('Reading %d ... %d  =  %9.3f ... %9.3f secs...' %
--> 566             (0, len(self.times) - 1, 0., self.times[-1]))
    567 self._data = self._read_segment(
    568     data_buffer=data_buffer, projector=self._projector)
    569 assert len(self._data) == self.info['nchan']

IndexError: index -1 is out of bounds for axis 0 with size 0
Additional information

sys_info:

Platform:         macOS-10.16-x86_64-i386-64bit
Python:           3.9.12 (main, Apr  5 2022, 01:53:17)  [Clang 12.0.0 ]
Executable:       /opt/anaconda3/envs/mne-bids-dev/bin/python
CPU:              i386: 4 cores
Memory:           Unavailable (requires "psutil" package)
mne:              1.0.3
numpy:            1.22.3 {}
scipy:            1.8.0
matplotlib:       3.5.2 {backend=MacOSX}

sklearn:          Not found
numba:            Not found
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           1.4.2
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
PyQt5:            Not found
ipympl:           Not found
pooch:            v1.6.0

mne_bids:         0.11.dev0
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the read_raw_brainvision function and RawBrainVision class, then reproduce the issue using the linked Archive.zip and the provided vhdr/eeg example. Check how the header's DataPoints value relates to the loaded binary data. Done means corrupted or insufficient binary content produces an informative error or warning instead of silently loading nonsense data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.