mne-tools / mne-tools/mne-python

Read_raw_eyelink failure: missing headpos data in file

Open
#12,516 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

BUG
Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

Description of the problem

A researcher reported a problem with our eyelink reader and shared the problematic Eyelink file (ASCII).

There are 2 separate issues.

  1. This file has an empty value for recording date/time, and our reader doesn't handle that gracefully. EDIT: The researcher actually manually removed the datetime from the ASCII file, so I think it's reasonable for us to continue assuming that EyeLink will write the datetime to the file.
  2. This file says that there are head-position (x, y, distance) data, but that data isn't actually in the file, and our reader doesn't handle that either.

I'll try to open a fix this week.

Steps to reproduce
from pathlib import Path
import mne

fname = Path(".") / "to" / "problem_file.asc"
mne.io.read_raw_eyelink(fname)
Link to data

See: https://mne.discourse.group/t/problem-reading-binocular-data-with-mne-io-read-raw-eyelink/8555

Expected results

successful file read

Actual results

BUG 1:

stack trace 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 raw = mne.io.read_raw_eyelink("example_data.asc")

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
     32 """Reader for an Eyelink ``.asc`` file.
     33 
     34 Parameters
   (...)
     58 ``'BAD_ACQ_SKIP'``.
     59 """
     60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname")
---> 62 raw_eyelink = RawEyelink(
     63     fname,
     64     create_annotations=create_annotations,
     65     apply_offsets=apply_offsets,
     66     find_overlaps=find_overlaps,
     67     overlap_threshold=overlap_threshold,
     68     verbose=verbose,
     69 )
     70 return raw_eyelink

File <decorator-gen-180>:12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
    104 fname = Path(fname)
    106 # ======================== Parse ASCII file ==========================
--> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
    108     fname, find_overlaps, overlap_threshold, apply_offsets
    109 )
    110 # ======================== Create Raw Object =========================
    111 super().__init__(
    112     info,
    113     preload=eye_ch_data,
   (...)
    116     raw_extras=[raw_extras],
    117 )

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:50, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets)
     48 raw_extras.update(_parse_recording_blocks(fname))
     49 raw_extras.update(_get_metadata(raw_extras))
---> 50 raw_extras["dt"] = _get_recording_datetime(fname)
     51 _validate_data(raw_extras)
     53 # ======================== Create DataFrames ========================

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:190, in _get_recording_datetime(fname)
    186 fmt = "%a %b %d %H:%M:%S %Y"
    187 # Eyelink measdate timestamps are timezone naive.
    188 # Force datetime to be in UTC.
    189 # Even though dt is probably in local time zone.
--> 190 dt_naive = datetime.strptime(dt_str, fmt)
    191 return dt_naive.replace(tzinfo=tz)

File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:568, in _strptime_datetime(cls, data_string, format)
    565 def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
    566     """Return a class cls instance based on the input string and the
    567     format string."""
--> 568     tt, fraction, gmtoff_fraction = _strptime(data_string, format)
    569     tzname, gmtoff = tt[-2:]
    570     args = tt[:6] + (fraction,)

File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:349, in _strptime(data_string, format)
    347 found = format_regex.match(data_string)
    348 if not found:
--> 349     raise ValueError("time data %r does not match format %r" %
    350                      (data_string, format))
    351 if len(data_string) != found.end():
    352     raise ValueError("unconverted data remains: %s" %
    353                       data_string[found.end():])

ValueError: time data '' does not match format '%a %b %d %H:%M:%S %Y'

BUG 2:

stack trace 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 raw = mne.io.read_raw_eyelink("example_data.asc")

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
     32 """Reader for an Eyelink ``.asc`` file.
     33 
     34 Parameters
   (...)
     58 ``'BAD_ACQ_SKIP'``.
     59 """
     60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname")
---> 62 raw_eyelink = RawEyelink(
     63     fname,
     64     create_annotations=create_annotations,
     65     apply_offsets=apply_offsets,
     66     find_overlaps=find_overlaps,
     67     overlap_threshold=overlap_threshold,
     68     verbose=verbose,
     69 )
     70 return raw_eyelink

File <decorator-gen-180>:12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
    104 fname = Path(fname)
    106 # ======================== Parse ASCII file ==========================
--> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
    108     fname, find_overlaps, overlap_threshold, apply_offsets
    109 )
    110 # ======================== Create Raw Object =========================
    111 super().__init__(
    112     info,
    113     preload=eye_ch_data,
   (...)
    116     raw_extras=[raw_extras],
    117 )

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:58, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets)
     56 # add column names to dataframes and set the dtype of each column
     57 col_names, ch_names = _infer_col_names(raw_extras)
---> 58 raw_extras["dfs"] = _assign_col_names(col_names, raw_extras["dfs"])
     59 raw_extras["dfs"] = _set_df_dtypes(raw_extras["dfs"])  # set dtypes for dataframes
     60 # if HREF data, convert to radians

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:407, in _assign_col_names(col_names, df_dict)
    405 for key, df in df_dict.items():
    406     if key in ("samples", "blinks", "fixations", "saccades"):
--> 407         df.columns = col_names[key]
    408     elif key == "messages":
    409         cols = ["time", "offset", "event_msg"]

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:6310, in NDFrame.__setattr__(self, name, value)
   6308 try:
   6309     object.__getattribute__(self, name)
-> 6310     return object.__setattr__(self, name, value)
   6311 except AttributeError:
   6312     pass

File properties.pyx:69, in pandas._libs.properties.AxisProperty.__set__()

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:813, in NDFrame._set_axis(self, axis, labels)
    808 """
    809 This is called from the cython code when we set the `index` attribute
    810 directly, e.g. `series.index = [1, 2, 3]`.
    811 """
    812 labels = ensure_index(labels)
--> 813 self._mgr.set_axis(axis, labels)
    814 self._clear_item_cache()

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/managers.py:238, in BaseBlockManager.set_axis(self, axis, new_labels)
    236 def set_axis(self, axis: AxisInt, new_labels: Index) -> None:
    237     # Caller is responsible for ensuring we have an Index object.
--> 238     self._validate_set_axis(axis, new_labels)
    239     self.axes[axis] = new_labels

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/base.py:98, in DataManager._validate_set_axis(self, axis, new_labels)
     95     pass
     97 elif new_len != old_len:
---> 98     raise ValueError(
     99         f"Length mismatch: Expected axis has {old_len} elements, new "
    100         f"values have {new_len} elements"
    101     )

ValueError: Length mismatch: Expected axis has 8 elements, new values have 11 elements
Additional information

N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at mne/io/eyelink/_utils.py, especially _get_recording_datetime, _parse_eyelink_ascii, and _assign_col_names, and reproduce the failure through mne.io.read_raw_eyelink using the reported ASCII file. The datetime case is withdrawn; done means the reader successfully loads files whose header advertises head-position data that is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.