mne-tools / mne-tools/mne-python
Researcher file breaks read_raw_eyelink
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 1.6k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 100
Description
Description of the problem
A researcher has reported that their file breaks read_raw_eyelink. I'm reporting the details below, so apologies in advance for the lengthy explanation:
Problem
In the specification for their ASCII format, EyeLink states that recording periods (where gaze/pupil data are actually being recorded) should always be demarcated by “START” and “END” lines, as shown below:
START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
END
For us, this is important because lines in the ASCII file that occur outside these START…END recording sections are unstructured and, IMO, are difficult to parse.
In other words, read_raw_eyelink looks for the START events, and parses lines until it hits an END event. Per Eyelink's specification, we always assume that any given START event will eventually be followed by an END event (before another START event occurs).
This assumption has held up until now. In the problematic file that the researcher shared, it looks like one of the recording blocks in the file is missing an “END” event, resulting in a format like:
START
xpos ypos pupil
xpos ypos pupil
END
...
START
xpos ypos pupil
xpos ypos pupil
...
START
xpos ypos pupil
xpos ypos pupil
END
So what happens is that for the block that is missing an END event, read_raw_eyelink tries to parse lines would typically occur outside recording blocks (specifically, these lines contain information about an eyetracking calibration), and thus that it is not prepared for. This breaks the reader.
I'm not sure how easy it will be to make our reader robust to this case. I might try some other EyeLink ASCII readers out there to see if they are able to read the file. In the mean time I'm opening this ticket so that we have a record of it.
Steps to reproduce
# Get the link to the file from the MNE forum (linked below)
from pathlib import Path
import mne
fname = Path().home() / "path" / "to" / "downloaded" / "file"
raw = mne.io.read_raw_eyelink(fname)
Link to data
https://drive.google.com/drive/folders/15SpQuoXZlmH6ZBLcOEoc4nzEA7ewoHuK
Expected results
a raw object
Actual results
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/eyelink.py", line 62, in read_raw_eyelink
raw_eyelink = RawEyelink(
File "<decorator-gen-202>", line 12, in __init__
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/eyelink.py", line 107, in __init__
eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/_utils.py", line 71, in _parse_eyelink_ascii
raw_extras["dfs"]["samples"] = _adjust_times(
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/mne/io/eyelink/_utils.py", line 509, in _adjust_times
return pd.merge_asof(
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 708, in merge_asof
return op.get_result()
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1926, in get_result
join_index, left_indexer, right_indexer = self._get_join_info()
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 1151, in _get_join_info
(left_indexer, right_indexer) = self._get_join_indexers()
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 2239, in _get_join_indexers
right_values = self._convert_values_for_libjoin(right_values, "right")
File "/Users/teichmanna2/anaconda3/envs/occ_beh/lib/python3.9/site-packages/pandas/core/reshape/merge.py", line 2182, in _convert_values_for_libjoin
raise ValueError(f"{side} keys must be sorted")
ValueError: right keys must be sorted
Additional information
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with mne/io/eyelink/_utils.py, especially _parse_eyelink_ascii and _adjust_times, and reproduce the failure with the linked EyeLink file using read_raw_eyelink. Investigate how a missing END before another START affects parsing and time adjustment; done means the supplied file loads into a raw object without the reported pandas sorting error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100