mne-tools / mne-tools/mne-python

How to handle datasets with invalid info[meas_id][secs]?

Open
#7,803 11 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

I'm woking with the ds000246 OpenNeuro dataset:

$ aws s3 sync --no-sign-request s3://openneuro.org/ds000246 ds000246
$ cd ds000246/sub-emptyroom/meg

Reading the data works as expected:

import mne
raw = mne.io.read_raw_ctf('sub-emptyroom_task-noise_run-01_meg.ds')

Writing thows an exception:

raw.save('/tmp/foo.fif')

Traceback:

RuntimeError                              Traceback (most recent call last)
<ipython-input-4-eb369e79ee42> in <module>
----> 1 raw.save('/tmp/foo.fif')

<decorator-gen-155> in save(self, fname, picks, tmin, tmax, buffer_size_sec, drop_small_buffer, proj, fmt, overwrite, split_size, split_naming, verbose)

~/Development/mne-python/mne/io/base.py in save(self, fname, picks, tmin, tmax, buffer_size_sec, drop_small_buffer, proj, fmt, overwrite, split_size, split_naming, verbose)
   1379                 "split_naming must be either 'neuromag' or 'bids' instead "
   1380                 "of '{}'.".format(split_naming))
-> 1381         _write_raw(fname, self, info, picks, fmt, data_type, reset_range,
   1382                    start, stop, buffer_size, projector, drop_small_buffer,
   1383                    split_size, split_naming, part_idx, None, overwrite)

~/Development/mne-python/mne/io/base.py in _write_raw(fname, raw, info, picks, fmt, data_type, reset_range, start, stop, buffer_size, projector, drop_small_buffer, split_size, split_naming, part_idx, prev_fname, overwrite)
   1844 
   1845     picks = _picks_to_idx(info, picks, 'all', ())
-> 1846     fid, cals = _start_writing_raw(use_fname, info, picks, data_type,
   1847                                    reset_range, raw.annotations)
   1848 

~/Development/mne-python/mne/io/base.py in _start_writing_raw(name, info, sel, data_type, reset_range, annotations)
   2018         cals.append(info['chs'][k]['cal'] * info['chs'][k]['range'])
   2019 
-> 2020     write_meas_info(fid, info, data_type=data_type, reset_range=reset_range)
   2021 
   2022     #

~/Development/mne-python/mne/io/meas_info.py in write_meas_info(fid, info, data_type, reset_range)
   1453     """
   1454     info._check_consistency()
-> 1455     _check_dates(info)
   1456 
   1457     # Measurement info

~/Development/mne-python/mne/io/meas_info.py in _check_dates(info, prepend_error)
   1411                 if (value[key_2] < np.iinfo('>i4').min or
   1412                         value[key_2] > np.iinfo('>i4').max):
-> 1413                     raise RuntimeError('%sinfo[%s][%s] must be between '
   1414                                        '"%r" and "%r", got "%r"'
   1415                                        % (prepend_error, key, key_2,

RuntimeError: info[meas_id][secs] must be between "-2147483648" and "2147483647", got "-5364633480"

How to best deal with data like this? Can I simply set info[meas_id][secs] to an arbitrary (valid) value? Also it seems a little odd that I can create (and work with) some data by reading it, but then cannot write it back to disk…

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the ds000246 CTF recording using mne.io.read_raw_ctf and raw.save. Read mne/io/base.py around _start_writing_raw and mne/io/meas_info.py around write_meas_info and _check_dates; done means agreeing on the supported handling for invalid info[meas_id][secs] and covering the read/write behavior with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.