ReceptiveField is not pipelineable

Nobody has claimed this yet.

Hi all,

I've been hitting a roadblock recently with the ReceptiveField module.

I'd like to use sklearn CV to find optimal hyperparameters for my data (typical dichotic listening experiment similar to the one in the tutorials :
https://mne-tools.github.io/dev/auto_tutorials/plot_receptive_field.html#compare-model-performance

In that example the crossvalidation and hyper parameter tuning is done "manually" in a for loop. This can be extremely slow as it does't use multiprocessing.

However when I tried using sklearn's GridSearchCV to find the best alpha, but I ran into all kinds of nasty sklearn reshaping errors. It's probably due to the input data format required by ReceptiveField (namely (n_epochs, n_times, n_chans)), which is:

not compatible with the rest of the MNE estimators like Scaler (which take (n_chans, n_epochs, n_times)
not compatible with sklearn's CV tools

For instance the code below will fail with a reshape-related error in sklearn :

from mne.decoding import Scaler
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
pipe = Pipeline([
    ('scaler', Scaler(scalings='mean')),
    ('rf', ReceptiveField(tmin, tmax, fs, scoring='r2',
                            feature_names=ch_names[:n_channels]))
    ])
kf = KFold(n_splits=n_splits, shuffle=False)
parameters = {
    'rf__estimator': np.logspace(0, 9, 10)
    }
rf = ReceptiveField(
    tmin, tmax, fs, feature_names=ch_names[:n_channels], scoring='r2')
grid = GridSearchCV(pipe, parameters, cv=kf, n_jobs=n_jobs,
                    scoring='r2', verbose=2)
grid.fit(eeg, y)
res = grid.cv_results_

...........................................................................
/Users/nicolas/anaconda3/envs/py3/lib/python3.6/site-packages/sklearn/metrics/regression.py in _check_reg_targets(y_true=array([[[-0.63683281],
        [-1.04570221],
  ...,
        [ 1.35630007],
        [ 0.39642897]]]), y_pred=array([[[-0.03673164],
        [-0.04872231],
  ...,
        [ 0.58824323],
        [ 0.1759169 ]]]), multioutput='uniform_average')
     71         just the corresponding argument if ``multioutput`` is a
     72         correct keyword.
     73
     74     """
     75     check_consistent_length(y_true, y_pred)
---> 76     y_true = check_array(y_true, ensure_2d=False)
        y_true = array([[[-0.63683281],
        [-1.04570221],
  ...,
        [ 1.35630007],
        [ 0.39642897]]])
     77     y_pred = check_array(y_pred, ensure_2d=False)
     78
     79     if y_true.ndim == 1:
     80         y_true = y_true.reshape((-1, 1))

...........................................................................
/Users/nicolas/anaconda3/envs/py3/lib/python3.6/site-packages/sklearn/utils/validation.py in check_array(array=array([[[-0.63683281],
        [-1.04570221],
  ...,
        [ 1.35630007],
        [ 0.39642897]]]), accept_sparse=False, dtype=None, order=None, copy=False, force_all_finite=True, ensure_2d=False, allow_nd=False, ensure_min_samples=1, ensure_min_features=1, warn_on_dtype=False, estimator=None)
    446         # make sure we actually converted to numeric:
    447         if dtype_numeric and array.dtype.kind == "O":
    448             array = array.astype(np.float64)
    449         if not allow_nd and array.ndim >= 3:
    450             raise ValueError("Found array with dim %d. %s expected <= 2."
--> 451                              % (array.ndim, estimator_name))
        array.ndim = 3
        estimator_name = 'Estimator'
    452         if force_all_finite:
    453             _assert_all_finite(array)
    454
    455     shape_repr = _shape_repr(array.shape)

ValueError: Found array with dim 3. Estimator expected <= 2.
___________________________________________________________________________

_

PS: actually I wanted to pass 'rf__estimator': [Ridge(alpha) for alpha in np.logspace(0, 9, 10)] but I'm not sure this is correct ?

Start with the ReceptiveField estimator and the sklearn Pipeline/GridSearchCV example in the issue, focusing on the documented input shapes and the reshaping error from sklearn metrics. Reproduce the failure with the provided code; done means ReceptiveField can participate in pipeline-based cross-validation and hyperparameter search without the reported dimensionality error.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment