mne-tools / mne-tools/mne-python

Fallback recovery for "EGI epoch first/last samps could not be parsed" (ValueError in read_raw_egi)

Open
#13,951 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

BUG
Dominant language
Python
Stars
3.5k
Forks
1.6k
Avg merge
1d 6h
Merged PRs (30d)
100

Description

Description of the problem

mne.io.read_raw_egi throws a ValueError when reading certain EGI .mff files
because n_samps_epochs (from epochs.xml) does not match the actual block samples in signal1.bin.

This mismatch occurs due to: Improperly closed recordings , asynchronous writes during PSG/PNS recording and pauses or impedance checks during recording that create irregular fractional epochs and maybe something else

I wrote a lightweight fallback function that reuses MNE's internal _get_blocks to calculate the exact duration based on the actual binary payload, recreating a valid epochs.xml on the fly - so n_samps_epochs == n_samps_block.

Would you be open to a Pull Request adding this as a fallback mechanism (e.g., recover_epochs=True) in read_raw_egi?

As a permanent solution within MNE, rather than rewriting the XML on disk,we can modifying the validation block in mne/io/egi/egimff.py (inside _read_header, (lines 97-108). Instead of raising a RuntimeError when bad == True, it could dynamically overwrite the epochs dictionary in memory with n_samps_block and signal_blocks["n_blocks"] (perhaps guarded by a recover_epochs=True parameter). So we can fix epoch file without altering the user's original raw data on disk

Thanks

Steps to reproduce
import xml.etree.ElementTree as ET
import re
import shutil
from pathlib import Path
from mne.io.egi.general import _get_blocks

def _recover_egi_epochs_xml(mff_path):
    """Recover or recreate a corrupted epochs.xml for EGI .mff files.

    This function parses the binary `signal1.bin` using MNE's internal `_get_blocks` 
    to calculate the exact number of blocks and samples. It then retrieves the time 
    divisor from `info.xml` and calculates the precise ``endTime`` in microseconds 
    to reconstruct a valid ``epochs.xml``.
    
    Parameters
    ----------
    mff_path : str | Path
        The path to the .mff directory.
    """
    mff_path = Path(mff_path)
    signal_bin = mff_path / "signal1.bin"
    epochs_xml = mff_path / "epochs.xml"
    info_xml_path = mff_path / "info.xml"
    
    if not signal_bin.exists() or not info_xml_path.exists():
        raise FileNotFoundError(f"Missing essential MFF files in {mff_path}")

    # 1. Use MNE's built-in parser to get exact block and sample counts
    # This guarantees our new epochs.xml will perfectly match MNE's expectations
    signal_blocks = _get_blocks(str(signal_bin))
    total_samples = int(signal_blocks["samples_block"].sum())
    last_block_idx = int(signal_blocks["n_blocks"])
    sfreq = int(signal_blocks["sfreq"])

    # 2. Extract record time format from info.xml to determine fractional multiplier
    tree = ET.parse(info_xml_path)
    root = tree.getroot()
    record_time_elem = root.find(".//{http://www.egi.com/info_mff}recordTime")
    if record_time_elem is None:
        raise ValueError("Could not find <recordTime> in info.xml")
        
    record_time = record_time_elem.text
    match = re.match(r".*\.(\d{6}(?:\d{3})?)[+-]", record_time)
    if not match:
        raise ValueError(f"Unexpected recordTime format: {record_time}")
        
    frac = match.group(1)
    # Determine if microseconds (len 6) or nanoseconds (len 9)
    div = 1000 if len(frac) == 6 else 1000000

    # 3. Calculate exact duration in microseconds
    # endTime is expected in microseconds.
    duration_us = int((total_samples / sfreq) * (div * 1000))

    # 4. Backup existing corrupted epochs.xml (if it exists)
    if epochs_xml.exists():
        backup_epochs_xml = mff_path.parent / f"{mff_path.name}__epochs_corrupted.xml"
        if not backup_epochs_xml.exists():
            shutil.copy2(epochs_xml, backup_epochs_xml)

    # 5. Write the corrected epochs.xml
    xml_str = (
        f'<?xml version="1.0" encoding="utf-8"?>\n'
        f'<epochs xmlns="http://www.egi.com/epochs_mff">\n'
        f'    <epoch>\n'
        f'        <beginTime>0</beginTime>\n'
        f'        <endTime>{duration_us}</endTime>\n'
        f'        <firstBlock>1</firstBlock>\n'
        f'        <lastBlock>{last_block_idx}</lastBlock>\n'
        f'    </epoch>\n'
        f'</epochs>\n'
    )
    epochs_xml.write_text(xml_str, encoding="utf-8")
    print(f"Successfully created/recovered epochs.xml for {mff_path.name} with {total_samples} samples and {last_block_idx} blocks.")
Link to data

No response

Expected results

MNE either loads the file (e.g. throwing a warning), or offers a recover_epochs=True parameter that uses its internal _get_blocks to rebuild the epochs.xml on the fly to salvage the clinical data.

Actual results

RuntimeError: EGI epoch first/last samps could not be parsed:
[0]
[17680358]

Additional information

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mne/io/egi/egimff.py, especially _read_header and the validation that raises the epoch parsing error, then inspect the _get_blocks entry point used with signal1.bin. Reproduce with an affected MFF file if available and compare epochs.xml against the binary block information. Done means read_raw_egi can handle the reported mismatch without altering the original data, with behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.