mne-tools / mne-tools/mne-python

Researcher file breaks read_raw_eyelink

Open
#12,690 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

BUG
Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

Description of the problem

A researcher has reported that their file breaks read_raw_eyelink. I'm reporting the details below, so apologies in advance for the lengthy explanation:

Problem

In the specification for their ASCII format, EyeLink states that recording periods (where gaze/pupil data are actually being recorded) should always be demarcated by “START” and “END” lines, as shown below:

START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
END

For us, this is important because lines in the ASCII file that occur outside these START…END recording sections are unstructured and, IMO, are difficult to parse.

In other words, read_raw_eyelink looks for the START events, and parses lines until it hits an END event. Per Eyelink's specification, we always assume that any given START event will eventually be followed by an END event (before another START event occurs).

This assumption has held up until now. In the problematic file that the researcher shared, it looks like one of the recording blocks in the file is missing an “END” event, resulting in a format like:

START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
...
START
xpos ypos pupil
xpos ypos pupil
END

So what happens is that for the block that is missing an END event, read_raw_eyelink tries to parse lines would typically occur outside recording blocks (specifically, these lines contain information about an eyetracking calibration), and thus that it is not prepared for. This breaks the reader.

I'm not sure how easy it will be to make our reader robust to this case. I might try some other EyeLink ASCII readers out there to see if they are able to read the file. In the mean time I'm opening this ticket so that we have a record of it.

Steps to reproduce
# Get the link to the file from the MNE forum (linked below)

from pathlib import Path
import mne

fname = Path().home() / "path" / "to" / "downloaded" / "file"

raw = mne.io.read_raw_eyelink(fname)
Link to data

https://drive.google.com/drive/folders/15SpQuoXZlmH6ZBLcOEoc4nzEA7ewoHuK

Expected results

a raw object

Actual results
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/eyelink.py", line 62, in read_raw_eyelink
    raw_eyelink = RawEyelink(
  File "<decorator-gen-202>", line 12, in __init__
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/eyelink.py", line 107, in __init__
    eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/_utils.py", line 71, in _parse_eyelink_ascii
    raw_extras["dfs"]["samples"] = _adjust_times(
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/_utils.py", line 509, in _adjust_times
    return pd.merge_asof(
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 708, in merge_asof
    return op.get_result()
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1926, in get_result
    join_index, left_indexer, right_indexer = self._get_join_info()
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1151, in _get_join_info
    (left_indexer, right_indexer) = self._get_join_indexers()
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 2239, in _get_join_indexers
    right_values = self._convert_values_for_libjoin(right_values, "right")
  File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 2182, in _convert_values_for_libjoin
    raise ValueError(f"{side} keys must be sorted")
ValueError: right keys must be sorted
Additional information

https://mne.discourse.group/t/mne-io-read-raw-eyelink-failure-adjust-times-sub-function-does-not-work-cant-merge/9012

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mne/io/eyelink/_utils.py, especially _parse_eyelink_ascii and _adjust_times, and reproduce the failure with the linked EyeLink file using read_raw_eyelink. Investigate how a missing END before another START affects parsing and time adjustment; done means the supplied file loads into a raw object without the reported pandas sorting error.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.