mne-tools / mne-tools/mne-python

raw.crop failure with multi-file CTF

Open
#6,482 10 comments 0 reactions 1 assignee View on GitHub

@massich is already working on this.

Since Jul 22, 2019.

Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

I'm unsure about the exact meaning of raw.first_samp, but I think there might be an error when cropping multi-file CTF recordings.

I'm working with ~1h long CTF recordings that I crop into blocks to preprocess. Unfortunately the CTF recordings is big (>10GB), so hard to share. The following demonstrates the problem:

>>> import mne
>>> raw = mne.io.read_raw_ctf(filename)
ds directory : /home/gortega/megdata/S1-5_Attractor_20161022_01.ds
    res4 data read.
    hc data read.
    Separate EEG position data file read.
    Quaternion matching (desired vs. transformed):
      -1.14   80.92    0.00 mm <->   -1.14   80.92    0.00 mm (orig :  -53.00   56.58 -262.06 mm) diff =    0.000 mm
       1.14  -80.92    0.00 mm <->    1.14  -80.92    0.00 mm (orig :   48.86  -69.11 -267.17 mm) diff =    0.000 mm
     100.30    0.00    0.00 mm <->  100.30    0.00    0.00 mm (orig :   75.61   54.06 -244.96 mm) diff =    0.000 mm
    Coordinate transformations established.
    Polhemus data for 3 HPI coils added
    Device coordinate locations for 3 HPI coils added
    64 EEG electrode locations assigned to channel info.
    64 EEG locations added to Polhemus data.
    Measurement info composed.
Finding samples for /home/gortega/megdata/S1-5_Attractor_20161022_01.ds/S1-5_Attractor_20161022_01.meg4: 
    System clock channel is available, checking which samples are valid.
    1104 x 1200 = 1324800 samples from 405 chs
Finding samples for /home/gortega/megdata/S1-5_Attractor_20161022_01.ds/S1-5_Attractor_20161022_01.1_meg4: 
    System clock channel is available, checking which samples are valid.
    1104 x 1200 = 1324800 samples from 405 chs
Finding samples for /home/gortega/megdata/S1-5_Attractor_20161022_01.ds/S1-5_Attractor_20161022_01.2_meg4: 
    System clock channel is available, checking which samples are valid.
    1104 x 1200 = 1324800 samples from 405 chs
Finding samples for /home/gortega/megdata/S1-5_Attractor_20161022_01.ds/S1-5_Attractor_20161022_01.3_meg4: 
    System clock channel is available, checking which samples are valid.
    1104 x 1200 = 1324800 samples from 405 chs
Finding samples for /home/gortega/megdata/S1-5_Attractor_20161022_01.ds/S1-5_Attractor_20161022_01.4_meg4: 
    System clock channel is available, checking which samples are valid.
    726 x 1200 = 871677 samples from 405 chs
    723 samples omitted at the end
Current compensation grade : 0

There are 5 meg4 files for this dataset and croping beyond the first meg4 file doesn't work:

# Crop within first meg4 file:
raw.copy().crop((100+1)/1200, (100+10)/1200).first_samp
101 # This is OK!
# Crop in 2nd meg4 file:
>>> raw.copy().crop((1324800+1)/1200, (1324800+10)/1200).first_samp
1 # I believe this should be 1324800+1

I think the source of the problem is that raw._first_samps and raw._last_samps are not set correctly:

>>> raw._first_samps
array([0, 0, 0, 0, 0])
>>> raw._last_samps
array([1324799, 1324799, 1324799, 1324799,  871676])

Fixing them like so solves the problem:

>>> raw._first_samps = np.cumsum(raw._raw_lengths) - raw._raw_lengths[0]
>>> raw._last_samps = np.cumsum(raw._last_samps)
>>>  raw.copy().crop((1324800+1)/1200, (1324800+10)/1200).first_samp
1324801 # OK!

But this leads to problems for me down the line.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.