Visualize bad sensors per trial

This example demonstrates how to use autoreject to visualize the bad sensors in each trial

# Author: Mainak Jas <mainak.jas@telecom-paristech.fr>
#         Denis A. Engemann <denis.engemann@gmail.com>
# License: BSD (3-clause)

# sphinx_gallery_thumbnail_number = 2

First, we download the data from OpenfMRI. We will download the tarfile, extract the necessary files and delete the tar from the disk

import os
import tarfile

import autoreject
from autoreject.utils import fetch_file

subject_id = 16  # OpenfMRI format of subject numbering

src_url = ('http://openfmri.s3.amazonaws.com/tarballs/'
           'ds117_R0.1.1_sub016_raw.tgz')
subject = "sub%03d" % subject_id

print("processing subject: %s" % subject)
base_path = os.path.join(
    os.path.dirname(autoreject.__file__), '..', 'examples')
target = os.path.join(base_path, 'ds117_R0.1.1_sub016_raw.tgz')
if not os.path.exists(os.path.join(base_path, 'ds117')):
    if not os.path.exists(target):
        fetch_file(src_url, target)
    tf = tarfile.open(target)
    print('Extracting files. This may take a while ...')
    tf.extractall(path=base_path, members=tf.getmembers()[-25:-9:3])
    os.remove(target)

Out:

processing subject: sub016

We will create epochs with data starting 200 ms before trigger onset and continuing up to 800 ms after that. The data contains visual stimuli for famous faces, unfamiliar faces, as well as scrambled faces.

tmin, tmax = -0.2, 0.8
events_id = {'famous/first': 5, 'famous/immediate': 6, 'famous/long': 7}

Let us now load all the epochs into memory and concatenate them

import mne  # noqa

epochs = list()
for run in range(3, 7):
    run_fname = os.path.join(base_path, 'ds117', 'sub%03d' % subject_id, 'MEG',
                             'run_%02d_raw.fif' % run)
    raw = mne.io.read_raw_fif(run_fname, preload=True)
    raw.pick_types(eeg=True, meg=False, stim=True)  # less memory + computation
    raw.filter(1., 40., l_trans_bandwidth=0.5, n_jobs=1, verbose='INFO')

    raw.set_channel_types({'EEG061': 'eog', 'EEG062': 'eog',
                           'EEG063': 'ecg', 'EEG064': 'misc'})
    raw.rename_channels({'EEG061': 'EOG061', 'EEG062': 'EOG062',
                         'EEG063': 'ECG063', 'EEG064': 'MISC'})

    events = mne.find_events(raw, stim_channel='STI101',
                             consecutive='increasing',
                             min_duration=0.003, verbose=True)
    # Read epochs
    mne.io.set_eeg_reference(raw)

    epoch = mne.Epochs(raw, events, events_id, tmin, tmax, proj=True,
                       baseline=None,
                       preload=False, reject=None, decim=4)
    epochs.append(epoch)

    # Same `dev_head_t` for all runs so that we can concatenate them.
    epoch.info['dev_head_t'] = epochs[0].info['dev_head_t']


epochs = mne.epochs.concatenate_epochs(epochs)

Out:

Opening raw data file /Users/mainak/Documents/github_repos/autoreject/autoreject/../examples/ds117/sub016/MEG/run_03_raw.fif...
    Read a total of 8 projection items:
        mag_ssp_upright.fif : PCA-mags-v1 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v2 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v3 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v4 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v5 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v1 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v2 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v3 (1 x 306)  idle
    Range : 60500 ... 598399 =     55.000 ...   543.999 secs
Ready.
Current compensation grade : 0
Reading 0 ... 537899  =      0.000 ...   488.999 secs...
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 1 - 40 Hz

FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 0.75 Hz)
- Upper passband edge: 40.00 Hz
- Upper transition bandwidth: 10.00 Hz (-6 dB cutoff frequency: 45.00 Hz)
- Filter length: 7261 samples (6.601 sec)

294 events found
Event IDs: [   5    6    7   13   14   15   17   18   19  256  261  262  263  269
  270  273  274  275 4096 4101 4102 4103 4109 4110 4111 4113 4114 4115]
EEG channel type selected for re-referencing
Applying average reference.
Applying a custom EEG reference.
49 matching events found
No baseline correction applied
Not setting metadata
8 projection items activated
Opening raw data file /Users/mainak/Documents/github_repos/autoreject/autoreject/../examples/ds117/sub016/MEG/run_04_raw.fif...
    Read a total of 8 projection items:
        mag_ssp_upright.fif : PCA-mags-v1 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v2 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v3 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v4 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v5 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v1 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v2 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v3 (1 x 306)  idle
    Range : 137500 ... 680899 =    125.000 ...   618.999 secs
Ready.
Current compensation grade : 0
Reading 0 ... 543399  =      0.000 ...   493.999 secs...
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 1 - 40 Hz

FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 0.75 Hz)
- Upper passband edge: 40.00 Hz
- Upper transition bandwidth: 10.00 Hz (-6 dB cutoff frequency: 45.00 Hz)
- Filter length: 7261 samples (6.601 sec)

296 events found
Event IDs: [   5    6    7   13   14   15   17   18   19  256  261  262  263  269
  270  273  274  275 4096 4101 4102 4103 4109 4110 4111 4113]
EEG channel type selected for re-referencing
Applying average reference.
Applying a custom EEG reference.
49 matching events found
No baseline correction applied
Not setting metadata
8 projection items activated
Opening raw data file /Users/mainak/Documents/github_repos/autoreject/autoreject/../examples/ds117/sub016/MEG/run_05_raw.fif...
    Read a total of 8 projection items:
        mag_ssp_upright.fif : PCA-mags-v1 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v2 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v3 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v4 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v5 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v1 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v2 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v3 (1 x 306)  idle
    Range : 279400 ... 820599 =    254.000 ...   745.999 secs
Ready.
Current compensation grade : 0
Reading 0 ... 541199  =      0.000 ...   491.999 secs...
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 1 - 40 Hz

FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 0.75 Hz)
- Upper passband edge: 40.00 Hz
- Upper transition bandwidth: 10.00 Hz (-6 dB cutoff frequency: 45.00 Hz)
- Filter length: 7261 samples (6.601 sec)

295 events found
Event IDs: [   5    6    7   13   14   15   17   18   19  256  261  262  263  269
  270  271  273  274  275 4096 4101 4102 4103 4109 4110 4111 4115 4357
 4359]
EEG channel type selected for re-referencing
Applying average reference.
Applying a custom EEG reference.
47 matching events found
No baseline correction applied
Not setting metadata
8 projection items activated
Opening raw data file /Users/mainak/Documents/github_repos/autoreject/autoreject/../examples/ds117/sub016/MEG/run_06_raw.fif...
    Read a total of 8 projection items:
        mag_ssp_upright.fif : PCA-mags-v1 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v2 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v3 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v4 (1 x 306)  idle
        mag_ssp_upright.fif : PCA-mags-v5 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v1 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v2 (1 x 306)  idle
        grad_ssp_upright.fif : PCA-grad-v3 (1 x 306)  idle
    Range : 140800 ... 680899 =    128.000 ...   618.999 secs
Ready.
Current compensation grade : 0
Reading 0 ... 540099  =      0.000 ...   490.999 secs...
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 1 - 40 Hz

FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 0.75 Hz)
- Upper passband edge: 40.00 Hz
- Upper transition bandwidth: 10.00 Hz (-6 dB cutoff frequency: 45.00 Hz)
- Filter length: 7261 samples (6.601 sec)

296 events found
Event IDs: [   5    6    7   13   14   15   17   18   19  256  261  262  263  269
  270  271  273  274  275 4096 4101 4102 4103 4109 4110 4111 4114]
EEG channel type selected for re-referencing
Applying average reference.
Applying a custom EEG reference.
48 matching events found
No baseline correction applied
Not setting metadata
8 projection items activated
Loading data for 49 events and 1101 original time points ...
0 bad epochs dropped
Loading data for 49 events and 1101 original time points ...
0 bad epochs dropped
Loading data for 47 events and 1101 original time points ...
0 bad epochs dropped
Loading data for 48 events and 1101 original time points ...
0 bad epochs dropped
193 matching events found
No baseline correction applied
Not setting metadata
0 bad epochs dropped

Now, we apply autoreject

from autoreject import AutoReject, compute_thresholds  # noqa

this_epoch = epochs['famous']
exclude = []  # XXX
picks = mne.pick_types(epochs.info, meg=False, eeg=True, stim=False,
                       eog=False, exclude=exclude)

Note that autoreject.AutoReject by design supports multiple channels. If no picks are passed separate solutions will be computed for each channel type and internally combines. This then readily supports cleaning unseen epochs from the different channel types used during fit. Here we only use a subset of channels to save time.

Also note that once the parameters are learned, any data can be repaired that contains channels that were used during fit. This also means that time may be saved by fitting autoreject.AutoReject on a representative subsample of the data.

ar = AutoReject(picks=picks, random_state=42, n_jobs=1, verbose='tqdm')

epochs_ar, reject_log = ar.fit_transform(this_epoch, return_log=True)

Out:

Running autoreject on ch_type=eeg

Creating augmented epochs:   0%|          | 0/70 [00:00<?, ?it/s]
Creating augmented epochs:   1%|1         | 1/70 [00:00<00:43,  1.57it/s]
Creating augmented epochs:   7%|7         | 5/70 [00:00<00:29,  2.20it/s]
Creating augmented epochs:  11%|#1        | 8/70 [00:00<00:20,  3.05it/s]
Creating augmented epochs:  17%|#7        | 12/70 [00:00<00:13,  4.18it/s]
Creating augmented epochs:  21%|##1       | 15/70 [00:01<00:09,  5.60it/s]
Creating augmented epochs:  26%|##5       | 18/70 [00:01<00:07,  7.34it/s]
Creating augmented epochs:  31%|###1      | 22/70 [00:01<00:05,  9.56it/s]
Creating augmented epochs:  37%|###7      | 26/70 [00:01<00:03, 12.23it/s]
Creating augmented epochs:  41%|####1     | 29/70 [00:01<00:02, 14.80it/s]
Creating augmented epochs:  47%|####7     | 33/70 [00:01<00:02, 17.68it/s]
Creating augmented epochs:  53%|#####2    | 37/70 [00:01<00:01, 19.10it/s]
Creating augmented epochs:  57%|#####7    | 40/70 [00:02<00:01, 19.02it/s]
Creating augmented epochs:  61%|######1   | 43/70 [00:02<00:01, 19.21it/s]
Creating augmented epochs:  66%|######5   | 46/70 [00:02<00:01, 20.45it/s]
Creating augmented epochs:  70%|#######   | 49/70 [00:02<00:01, 20.50it/s]
Creating augmented epochs:  74%|#######4  | 52/70 [00:02<00:00, 21.61it/s]
Creating augmented epochs:  79%|#######8  | 55/70 [00:02<00:00, 22.59it/s]
Creating augmented epochs:  83%|########2 | 58/70 [00:02<00:00, 23.19it/s]
Creating augmented epochs:  87%|########7 | 61/70 [00:02<00:00, 23.91it/s]
Creating augmented epochs:  91%|#########1| 64/70 [00:03<00:00, 19.73it/s]
Creating augmented epochs:  96%|#########5| 67/70 [00:03<00:00, 21.61it/s]
Creating augmented epochs: 100%|##########| 70/70 [00:03<00:00, 21.09it/s]

Computing thresholds ...:   0%|          | 0/70 [00:00<?, ?it/s]
Computing thresholds ...:   1%|1         | 1/70 [00:01<02:14,  1.94s/it]
Computing thresholds ...:   3%|2         | 2/70 [00:03<02:13,  1.96s/it]
Computing thresholds ...:   4%|4         | 3/70 [00:05<02:00,  1.80s/it]
Computing thresholds ...:   6%|5         | 4/70 [00:06<01:52,  1.71s/it]
Computing thresholds ...:   7%|7         | 5/70 [00:08<01:51,  1.71s/it]
Computing thresholds ...:   9%|8         | 6/70 [00:10<01:49,  1.70s/it]
Computing thresholds ...:  10%|#         | 7/70 [00:11<01:38,  1.57s/it]
Computing thresholds ...:  11%|#1        | 8/70 [00:12<01:33,  1.52s/it]
Computing thresholds ...:  13%|#2        | 9/70 [00:14<01:35,  1.57s/it]
Computing thresholds ...:  14%|#4        | 10/70 [00:16<01:37,  1.62s/it]
Computing thresholds ...:  16%|#5        | 11/70 [00:18<01:46,  1.81s/it]
Computing thresholds ...:  17%|#7        | 12/70 [00:19<01:36,  1.67s/it]
Computing thresholds ...:  19%|#8        | 13/70 [00:21<01:29,  1.58s/it]
Computing thresholds ...:  20%|##        | 14/70 [00:22<01:24,  1.52s/it]
Computing thresholds ...:  21%|##1       | 15/70 [00:23<01:20,  1.46s/it]
Computing thresholds ...:  23%|##2       | 16/70 [00:25<01:17,  1.44s/it]
Computing thresholds ...:  24%|##4       | 17/70 [00:26<01:14,  1.40s/it]
Computing thresholds ...:  26%|##5       | 18/70 [00:28<01:13,  1.41s/it]
Computing thresholds ...:  27%|##7       | 19/70 [00:29<01:11,  1.39s/it]
Computing thresholds ...:  29%|##8       | 20/70 [00:30<01:09,  1.38s/it]
Computing thresholds ...:  30%|###       | 21/70 [00:32<01:05,  1.35s/it]
Computing thresholds ...:  31%|###1      | 22/70 [00:33<01:06,  1.39s/it]
Computing thresholds ...:  33%|###2      | 23/70 [00:35<01:06,  1.42s/it]
Computing thresholds ...:  34%|###4      | 24/70 [00:36<01:04,  1.40s/it]
Computing thresholds ...:  36%|###5      | 25/70 [00:37<01:00,  1.34s/it]
Computing thresholds ...:  37%|###7      | 26/70 [00:39<01:01,  1.41s/it]
Computing thresholds ...:  39%|###8      | 27/70 [00:41<01:16,  1.79s/it]
Computing thresholds ...:  40%|####      | 28/70 [00:43<01:13,  1.75s/it]
Computing thresholds ...:  41%|####1     | 29/70 [00:45<01:14,  1.81s/it]
Computing thresholds ...:  43%|####2     | 30/70 [00:47<01:09,  1.74s/it]
Computing thresholds ...:  44%|####4     | 31/70 [00:48<01:02,  1.61s/it]
Computing thresholds ...:  46%|####5     | 32/70 [00:50<01:04,  1.68s/it]
Computing thresholds ...:  47%|####7     | 33/70 [00:51<00:59,  1.61s/it]
Computing thresholds ...:  49%|####8     | 34/70 [00:53<00:57,  1.60s/it]
Computing thresholds ...:  50%|#####     | 35/70 [00:55<01:04,  1.86s/it]
Computing thresholds ...:  51%|#####1    | 36/70 [00:56<00:55,  1.63s/it]
Computing thresholds ...:  53%|#####2    | 37/70 [00:58<00:50,  1.53s/it]
Computing thresholds ...:  54%|#####4    | 38/70 [00:59<00:46,  1.46s/it]
Computing thresholds ...:  56%|#####5    | 39/70 [01:00<00:46,  1.49s/it]
Computing thresholds ...:  57%|#####7    | 40/70 [01:02<00:43,  1.45s/it]
Computing thresholds ...:  59%|#####8    | 41/70 [01:03<00:39,  1.36s/it]
Computing thresholds ...:  60%|######    | 42/70 [01:04<00:38,  1.37s/it]
Computing thresholds ...:  61%|######1   | 43/70 [01:06<00:36,  1.35s/it]
Computing thresholds ...:  63%|######2   | 44/70 [01:07<00:33,  1.30s/it]
Computing thresholds ...:  64%|######4   | 45/70 [01:08<00:33,  1.34s/it]
Computing thresholds ...:  66%|######5   | 46/70 [01:09<00:31,  1.30s/it]
Computing thresholds ...:  67%|######7   | 47/70 [01:11<00:32,  1.41s/it]
Computing thresholds ...:  69%|######8   | 48/70 [01:14<00:40,  1.82s/it]
Computing thresholds ...:  70%|#######   | 49/70 [01:20<01:01,  2.95s/it]
Computing thresholds ...:  71%|#######1  | 50/70 [01:23<01:04,  3.22s/it]
Computing thresholds ...:  73%|#######2  | 51/70 [01:25<00:53,  2.80s/it]
Computing thresholds ...:  74%|#######4  | 52/70 [01:27<00:44,  2.45s/it]
Computing thresholds ...:  76%|#######5  | 53/70 [01:28<00:35,  2.11s/it]
Computing thresholds ...:  77%|#######7  | 54/70 [01:29<00:29,  1.84s/it]
Computing thresholds ...:  79%|#######8  | 55/70 [01:31<00:24,  1.66s/it]
Computing thresholds ...:  80%|########  | 56/70 [01:32<00:22,  1.57s/it]
Computing thresholds ...:  81%|########1 | 57/70 [01:33<00:19,  1.51s/it]
Computing thresholds ...:  83%|########2 | 58/70 [01:35<00:17,  1.47s/it]
Computing thresholds ...:  84%|########4 | 59/70 [01:36<00:16,  1.46s/it]
Computing thresholds ...:  86%|########5 | 60/70 [01:37<00:13,  1.37s/it]
Computing thresholds ...:  87%|########7 | 61/70 [01:39<00:11,  1.33s/it]
Computing thresholds ...:  89%|########8 | 62/70 [01:40<00:10,  1.32s/it]
Computing thresholds ...:  90%|######### | 63/70 [01:41<00:08,  1.28s/it]
Computing thresholds ...:  91%|#########1| 64/70 [01:42<00:07,  1.29s/it]
Computing thresholds ...:  93%|#########2| 65/70 [01:44<00:06,  1.27s/it]
Computing thresholds ...:  94%|#########4| 66/70 [01:45<00:05,  1.29s/it]
Computing thresholds ...:  96%|#########5| 67/70 [01:46<00:03,  1.26s/it]
Computing thresholds ...:  97%|#########7| 68/70 [01:47<00:02,  1.22s/it]
Computing thresholds ...:  99%|#########8| 69/70 [01:49<00:01,  1.25s/it]
Computing thresholds ...: 100%|##########| 70/70 [01:50<00:00,  1.22s/it]



Repairing epochs:   0%|          | 0/193 [00:00<?, ?it/s]


Repairing epochs:   5%|4         | 9/193 [00:00<00:02, 83.98it/s]


Repairing epochs:  10%|9         | 19/193 [00:00<00:02, 86.12it/s]


Repairing epochs:  15%|#4        | 28/193 [00:00<00:01, 86.44it/s]


Repairing epochs:  19%|#9        | 37/193 [00:00<00:01, 86.25it/s]


Repairing epochs:  24%|##3       | 46/193 [00:00<00:01, 86.87it/s]


Repairing epochs:  28%|##8       | 55/193 [00:00<00:01, 87.59it/s]


Repairing epochs:  34%|###3      | 65/193 [00:00<00:01, 88.45it/s]


Repairing epochs:  38%|###8      | 74/193 [00:00<00:01, 88.89it/s]


Repairing epochs:  44%|####3     | 84/193 [00:00<00:01, 89.88it/s]


Repairing epochs:  49%|####8     | 94/193 [00:01<00:01, 89.67it/s]


Repairing epochs:  53%|#####3    | 103/193 [00:01<00:01, 87.92it/s]


Repairing epochs:  58%|#####8    | 112/193 [00:01<00:00, 86.39it/s]


Repairing epochs:  63%|######3   | 122/193 [00:01<00:00, 87.94it/s]


Repairing epochs:  68%|######7   | 131/193 [00:01<00:00, 87.60it/s]


Repairing epochs:  73%|#######3  | 141/193 [00:01<00:00, 88.40it/s]


Repairing epochs:  78%|#######7  | 150/193 [00:01<00:00, 88.53it/s]


Repairing epochs:  83%|########2 | 160/193 [00:01<00:00, 89.13it/s]


Repairing epochs:  88%|########8 | 170/193 [00:01<00:00, 89.95it/s]


Repairing epochs:  93%|#########2| 179/193 [00:02<00:00, 78.24it/s]


Repairing epochs:  97%|#########7| 188/193 [00:02<00:00, 79.67it/s]


Repairing epochs: 100%|##########| 193/193 [00:02<00:00, 86.55it/s]

n_interp:   0%|          | 0/3 [00:00<?, ?it/s]


Repairing epochs:   0%|          | 0/193 [00:00<?, ?it/s]


Repairing epochs:   2%|2         | 4/193 [00:00<00:05, 35.88it/s]


Repairing epochs:   4%|4         | 8/193 [00:00<00:05, 34.95it/s]


Repairing epochs:   6%|6         | 12/193 [00:00<00:05, 34.51it/s]


Repairing epochs:   8%|8         | 16/193 [00:00<00:05, 35.11it/s]


Repairing epochs:  10%|#         | 20/193 [00:00<00:04, 34.86it/s]


Repairing epochs:  12%|#2        | 24/193 [00:00<00:04, 34.74it/s]


Repairing epochs:  15%|#4        | 28/193 [00:00<00:04, 35.49it/s]


Repairing epochs:  17%|#6        | 32/193 [00:00<00:04, 35.46it/s]


Repairing epochs:  19%|#8        | 36/193 [00:01<00:04, 35.56it/s]


Repairing epochs:  21%|##        | 40/193 [00:01<00:04, 35.11it/s]


Repairing epochs:  23%|##2       | 44/193 [00:01<00:04, 36.07it/s]


Repairing epochs:  25%|##4       | 48/193 [00:01<00:04, 35.06it/s]


Repairing epochs:  27%|##6       | 52/193 [00:01<00:04, 34.92it/s]


Repairing epochs:  29%|##9       | 56/193 [00:01<00:03, 34.71it/s]


Repairing epochs:  31%|###1      | 60/193 [00:01<00:03, 35.10it/s]


Repairing epochs:  33%|###3      | 64/193 [00:01<00:03, 35.26it/s]


Repairing epochs:  35%|###5      | 68/193 [00:01<00:03, 35.55it/s]


Repairing epochs:  37%|###7      | 72/193 [00:02<00:03, 35.96it/s]


Repairing epochs:  39%|###9      | 76/193 [00:02<00:03, 35.81it/s]


Repairing epochs:  41%|####1     | 80/193 [00:02<00:03, 36.30it/s]


Repairing epochs:  44%|####3     | 84/193 [00:02<00:03, 35.36it/s]


Repairing epochs:  46%|####5     | 88/193 [00:02<00:03, 34.94it/s]


Repairing epochs:  48%|####7     | 92/193 [00:02<00:02, 35.31it/s]


Repairing epochs:  50%|####9     | 96/193 [00:02<00:02, 34.51it/s]


Repairing epochs:  52%|#####1    | 100/193 [00:02<00:02, 35.45it/s]


Repairing epochs:  54%|#####3    | 104/193 [00:02<00:02, 35.96it/s]


Repairing epochs:  56%|#####5    | 108/193 [00:03<00:02, 35.30it/s]


Repairing epochs:  58%|#####8    | 112/193 [00:03<00:02, 35.17it/s]


Repairing epochs:  60%|######    | 116/193 [00:03<00:02, 34.78it/s]


Repairing epochs:  62%|######2   | 120/193 [00:03<00:02, 34.90it/s]


Repairing epochs:  64%|######4   | 124/193 [00:03<00:01, 34.97it/s]


Repairing epochs:  66%|######6   | 128/193 [00:03<00:02, 29.67it/s]


Repairing epochs:  68%|######8   | 132/193 [00:03<00:01, 31.30it/s]


Repairing epochs:  70%|#######   | 136/193 [00:03<00:01, 32.22it/s]


Repairing epochs:  73%|#######2  | 140/193 [00:04<00:01, 32.36it/s]


Repairing epochs:  75%|#######4  | 144/193 [00:04<00:01, 31.59it/s]


Repairing epochs:  77%|#######6  | 148/193 [00:04<00:01, 30.84it/s]


Repairing epochs:  79%|#######8  | 152/193 [00:04<00:01, 30.63it/s]


Repairing epochs:  81%|########  | 156/193 [00:04<00:01, 31.44it/s]


Repairing epochs:  83%|########2 | 160/193 [00:04<00:01, 32.95it/s]


Repairing epochs:  85%|########4 | 164/193 [00:04<00:00, 32.17it/s]


Repairing epochs:  87%|########7 | 168/193 [00:04<00:00, 30.43it/s]


Repairing epochs:  89%|########9 | 172/193 [00:05<00:00, 28.51it/s]


Repairing epochs:  91%|#########1| 176/193 [00:05<00:00, 29.76it/s]


Repairing epochs:  93%|#########3| 180/193 [00:05<00:00, 30.97it/s]


Repairing epochs:  96%|#########5| 185/193 [00:05<00:00, 32.94it/s]


Repairing epochs:  98%|#########7| 189/193 [00:05<00:00, 33.59it/s]


Repairing epochs: 100%|##########| 193/193 [00:05<00:00, 33.50it/s]



Fold:   0%|          | 0/10 [00:00<?, ?it/s]



Fold:  10%|#         | 1/10 [00:00<00:05,  1.63it/s]



Fold:  20%|##        | 2/10 [00:01<00:04,  1.67it/s]



Fold:  30%|###       | 3/10 [00:01<00:04,  1.71it/s]



Fold:  40%|####      | 4/10 [00:02<00:03,  1.71it/s]



Fold:  50%|#####     | 5/10 [00:02<00:02,  1.73it/s]



Fold:  60%|######    | 6/10 [00:03<00:02,  1.76it/s]



Fold:  70%|#######   | 7/10 [00:03<00:01,  1.78it/s]



Fold:  80%|########  | 8/10 [00:04<00:01,  1.79it/s]



Fold:  90%|######### | 9/10 [00:05<00:00,  1.80it/s]



Fold: 100%|##########| 10/10 [00:05<00:00,  1.81it/s]

n_interp:  33%|###3      | 1/3 [00:13<00:26, 13.31s/it]


Repairing epochs:   0%|          | 0/193 [00:00<?, ?it/s]


Repairing epochs:   2%|2         | 4/193 [00:00<00:06, 30.76it/s]


Repairing epochs:   4%|4         | 8/193 [00:00<00:05, 31.57it/s]


Repairing epochs:   6%|6         | 12/193 [00:00<00:05, 30.98it/s]


Repairing epochs:   8%|7         | 15/193 [00:00<00:05, 30.38it/s]


Repairing epochs:  10%|9         | 19/193 [00:00<00:05, 31.21it/s]


Repairing epochs:  12%|#1        | 23/193 [00:00<00:05, 32.63it/s]


Repairing epochs:  14%|#3        | 27/193 [00:00<00:05, 32.98it/s]


Repairing epochs:  16%|#6        | 31/193 [00:00<00:04, 33.65it/s]


Repairing epochs:  18%|#8        | 35/193 [00:01<00:04, 34.02it/s]


Repairing epochs:  20%|##        | 39/193 [00:01<00:04, 34.69it/s]


Repairing epochs:  22%|##2       | 43/193 [00:01<00:04, 34.12it/s]


Repairing epochs:  24%|##4       | 47/193 [00:01<00:04, 34.36it/s]


Repairing epochs:  26%|##6       | 51/193 [00:01<00:04, 34.07it/s]


Repairing epochs:  28%|##8       | 55/193 [00:01<00:03, 34.56it/s]


Repairing epochs:  31%|###       | 59/193 [00:01<00:03, 34.86it/s]


Repairing epochs:  33%|###2      | 63/193 [00:01<00:03, 35.41it/s]


Repairing epochs:  35%|###4      | 67/193 [00:02<00:04, 29.84it/s]


Repairing epochs:  37%|###6      | 71/193 [00:02<00:03, 31.76it/s]


Repairing epochs:  39%|###8      | 75/193 [00:02<00:03, 32.78it/s]


Repairing epochs:  41%|####      | 79/193 [00:02<00:03, 32.63it/s]


Repairing epochs:  43%|####3     | 83/193 [00:02<00:03, 33.06it/s]


Repairing epochs:  45%|####5     | 87/193 [00:02<00:03, 33.03it/s]


Repairing epochs:  47%|####7     | 91/193 [00:02<00:03, 33.59it/s]


Repairing epochs:  49%|####9     | 95/193 [00:02<00:02, 33.68it/s]


Repairing epochs:  51%|#####1    | 99/193 [00:02<00:02, 33.75it/s]


Repairing epochs:  53%|#####3    | 103/193 [00:03<00:02, 33.70it/s]


Repairing epochs:  55%|#####5    | 107/193 [00:03<00:02, 33.75it/s]


Repairing epochs:  58%|#####7    | 111/193 [00:03<00:02, 33.87it/s]


Repairing epochs:  60%|#####9    | 115/193 [00:03<00:02, 33.93it/s]


Repairing epochs:  62%|######1   | 119/193 [00:03<00:02, 34.41it/s]


Repairing epochs:  64%|######3   | 123/193 [00:03<00:02, 34.38it/s]


Repairing epochs:  66%|######5   | 127/193 [00:03<00:01, 34.50it/s]


Repairing epochs:  68%|######7   | 131/193 [00:03<00:01, 33.95it/s]


Repairing epochs:  70%|######9   | 135/193 [00:04<00:01, 33.47it/s]


Repairing epochs:  72%|#######2  | 139/193 [00:04<00:01, 34.02it/s]


Repairing epochs:  74%|#######4  | 143/193 [00:04<00:01, 34.31it/s]


Repairing epochs:  76%|#######6  | 147/193 [00:04<00:01, 34.18it/s]


Repairing epochs:  78%|#######8  | 151/193 [00:04<00:01, 34.39it/s]


Repairing epochs:  80%|########  | 155/193 [00:04<00:01, 33.96it/s]


Repairing epochs:  82%|########2 | 159/193 [00:04<00:01, 33.88it/s]


Repairing epochs:  84%|########4 | 163/193 [00:04<00:00, 34.53it/s]


Repairing epochs:  87%|########6 | 167/193 [00:04<00:00, 34.40it/s]


Repairing epochs:  89%|########8 | 171/193 [00:05<00:00, 34.23it/s]


Repairing epochs:  91%|######### | 175/193 [00:05<00:00, 34.06it/s]


Repairing epochs:  93%|#########2| 179/193 [00:05<00:00, 33.43it/s]


Repairing epochs:  95%|#########4| 183/193 [00:05<00:00, 34.54it/s]


Repairing epochs:  97%|#########6| 187/193 [00:05<00:00, 33.69it/s]


Repairing epochs:  99%|#########8| 191/193 [00:05<00:00, 33.61it/s]


Repairing epochs: 100%|##########| 193/193 [00:05<00:00, 33.61it/s]



Fold:   0%|          | 0/10 [00:00<?, ?it/s]



Fold:  10%|#         | 1/10 [00:00<00:05,  1.70it/s]



Fold:  20%|##        | 2/10 [00:01<00:04,  1.73it/s]



Fold:  30%|###       | 3/10 [00:01<00:03,  1.76it/s]



Fold:  40%|####      | 4/10 [00:02<00:03,  1.78it/s]



Fold:  50%|#####     | 5/10 [00:02<00:02,  1.74it/s]



Fold:  60%|######    | 6/10 [00:03<00:02,  1.72it/s]



Fold:  70%|#######   | 7/10 [00:03<00:01,  1.76it/s]



Fold:  80%|########  | 8/10 [00:04<00:01,  1.73it/s]



Fold:  90%|######### | 9/10 [00:05<00:00,  1.76it/s]



Fold: 100%|##########| 10/10 [00:05<00:00,  1.78it/s]

n_interp:  67%|######6   | 2/3 [00:26<00:13, 13.13s/it]


Repairing epochs:   0%|          | 0/193 [00:00<?, ?it/s]


Repairing epochs:   2%|2         | 4/193 [00:00<00:05, 32.91it/s]


Repairing epochs:   4%|4         | 8/193 [00:00<00:05, 32.88it/s]


Repairing epochs:   6%|6         | 12/193 [00:00<00:05, 32.59it/s]


Repairing epochs:   8%|8         | 16/193 [00:00<00:05, 33.33it/s]


Repairing epochs:  10%|#         | 20/193 [00:00<00:05, 31.85it/s]


Repairing epochs:  12%|#1        | 23/193 [00:00<00:05, 31.19it/s]


Repairing epochs:  14%|#3        | 27/193 [00:00<00:05, 32.25it/s]


Repairing epochs:  16%|#6        | 31/193 [00:00<00:04, 33.74it/s]


Repairing epochs:  18%|#8        | 35/193 [00:01<00:04, 34.10it/s]


Repairing epochs:  20%|##        | 39/193 [00:01<00:04, 34.41it/s]


Repairing epochs:  22%|##2       | 43/193 [00:01<00:05, 28.91it/s]


Repairing epochs:  24%|##4       | 47/193 [00:01<00:04, 30.20it/s]


Repairing epochs:  26%|##6       | 51/193 [00:01<00:04, 31.70it/s]


Repairing epochs:  28%|##8       | 55/193 [00:01<00:04, 33.04it/s]


Repairing epochs:  31%|###       | 59/193 [00:01<00:04, 32.77it/s]


Repairing epochs:  33%|###2      | 63/193 [00:01<00:03, 33.47it/s]


Repairing epochs:  35%|###4      | 67/193 [00:02<00:03, 33.22it/s]


Repairing epochs:  37%|###6      | 71/193 [00:02<00:03, 33.55it/s]


Repairing epochs:  39%|###8      | 75/193 [00:02<00:03, 34.34it/s]


Repairing epochs:  41%|####      | 79/193 [00:02<00:03, 35.03it/s]


Repairing epochs:  43%|####3     | 83/193 [00:02<00:03, 35.05it/s]


Repairing epochs:  45%|####5     | 87/193 [00:02<00:03, 33.90it/s]


Repairing epochs:  47%|####7     | 91/193 [00:02<00:03, 33.90it/s]


Repairing epochs:  49%|####9     | 95/193 [00:02<00:02, 34.57it/s]


Repairing epochs:  51%|#####1    | 99/193 [00:02<00:02, 33.17it/s]


Repairing epochs:  53%|#####3    | 103/193 [00:03<00:02, 33.74it/s]


Repairing epochs:  55%|#####5    | 107/193 [00:03<00:02, 34.10it/s]


Repairing epochs:  58%|#####7    | 111/193 [00:03<00:02, 35.16it/s]


Repairing epochs:  60%|#####9    | 115/193 [00:03<00:02, 34.61it/s]


Repairing epochs:  62%|######1   | 119/193 [00:03<00:02, 34.69it/s]


Repairing epochs:  64%|######3   | 123/193 [00:03<00:02, 34.63it/s]


Repairing epochs:  66%|######5   | 127/193 [00:03<00:01, 33.29it/s]


Repairing epochs:  68%|######7   | 131/193 [00:03<00:01, 33.09it/s]


Repairing epochs:  70%|######9   | 135/193 [00:04<00:01, 34.63it/s]


Repairing epochs:  72%|#######2  | 139/193 [00:04<00:01, 34.41it/s]


Repairing epochs:  74%|#######4  | 143/193 [00:04<00:01, 35.09it/s]


Repairing epochs:  76%|#######6  | 147/193 [00:04<00:01, 35.51it/s]


Repairing epochs:  78%|#######8  | 151/193 [00:04<00:01, 36.14it/s]


Repairing epochs:  81%|########  | 156/193 [00:04<00:00, 37.50it/s]


Repairing epochs:  83%|########2 | 160/193 [00:04<00:00, 34.50it/s]


Repairing epochs:  85%|########4 | 164/193 [00:04<00:00, 34.46it/s]


Repairing epochs:  87%|########7 | 168/193 [00:04<00:00, 33.36it/s]


Repairing epochs:  89%|########9 | 172/193 [00:05<00:00, 34.24it/s]


Repairing epochs:  91%|#########1| 176/193 [00:05<00:00, 34.33it/s]


Repairing epochs:  93%|#########3| 180/193 [00:05<00:00, 33.64it/s]


Repairing epochs:  95%|#########5| 184/193 [00:05<00:00, 33.13it/s]


Repairing epochs:  97%|#########7| 188/193 [00:05<00:00, 30.90it/s]


Repairing epochs:  99%|#########9| 192/193 [00:05<00:00, 29.20it/s]


Repairing epochs: 100%|##########| 193/193 [00:05<00:00, 33.25it/s]



Fold:   0%|          | 0/10 [00:00<?, ?it/s]



Fold:  10%|#         | 1/10 [00:00<00:03,  2.70it/s]



Fold:  20%|##        | 2/10 [00:00<00:02,  2.77it/s]



Fold:  30%|###       | 3/10 [00:01<00:02,  2.81it/s]



Fold:  40%|####      | 4/10 [00:01<00:02,  2.84it/s]



Fold:  50%|#####     | 5/10 [00:01<00:01,  2.89it/s]



Fold:  60%|######    | 6/10 [00:02<00:01,  2.93it/s]



Fold:  70%|#######   | 7/10 [00:02<00:01,  2.92it/s]



Fold:  80%|########  | 8/10 [00:02<00:00,  2.85it/s]



Fold:  90%|######### | 9/10 [00:03<00:00,  2.87it/s]



Fold: 100%|##########| 10/10 [00:03<00:00,  2.92it/s]

n_interp: 100%|##########| 3/3 [00:35<00:00, 12.05s/it]



Estimated consensus=0.10 and n_interpolate=4



Repairing epochs:   0%|          | 0/193 [00:00<?, ?it/s]


Repairing epochs:   2%|2         | 4/193 [00:00<00:05, 34.84it/s]


Repairing epochs:   4%|4         | 8/193 [00:00<00:05, 34.10it/s]


Repairing epochs:   5%|5         | 10/193 [00:00<00:06, 26.47it/s]


Repairing epochs:   7%|7         | 14/193 [00:00<00:06, 28.67it/s]


Repairing epochs:   9%|9         | 18/193 [00:00<00:05, 30.27it/s]


Repairing epochs:  11%|#1        | 22/193 [00:00<00:05, 32.30it/s]


Repairing epochs:  13%|#3        | 26/193 [00:00<00:05, 32.64it/s]


Repairing epochs:  16%|#5        | 30/193 [00:00<00:04, 33.65it/s]


Repairing epochs:  18%|#7        | 34/193 [00:01<00:04, 33.94it/s]


Repairing epochs:  20%|#9        | 38/193 [00:01<00:04, 34.19it/s]


Repairing epochs:  22%|##1       | 42/193 [00:01<00:04, 33.64it/s]


Repairing epochs:  24%|##3       | 46/193 [00:01<00:04, 34.30it/s]


Repairing epochs:  26%|##5       | 50/193 [00:01<00:04, 34.08it/s]


Repairing epochs:  28%|##7       | 54/193 [00:01<00:04, 34.56it/s]


Repairing epochs:  30%|###       | 58/193 [00:01<00:03, 34.95it/s]


Repairing epochs:  32%|###2      | 62/193 [00:01<00:03, 35.33it/s]


Repairing epochs:  34%|###4      | 66/193 [00:01<00:03, 34.28it/s]


Repairing epochs:  36%|###6      | 70/193 [00:02<00:03, 33.19it/s]


Repairing epochs:  38%|###8      | 74/193 [00:02<00:03, 33.61it/s]


Repairing epochs:  40%|####      | 78/193 [00:02<00:03, 33.72it/s]


Repairing epochs:  42%|####2     | 82/193 [00:02<00:03, 32.99it/s]


Repairing epochs:  45%|####4     | 86/193 [00:02<00:03, 31.23it/s]


Repairing epochs:  47%|####6     | 90/193 [00:02<00:03, 31.50it/s]


Repairing epochs:  49%|####8     | 94/193 [00:02<00:03, 32.17it/s]


Repairing epochs:  51%|#####     | 98/193 [00:02<00:02, 32.84it/s]


Repairing epochs:  53%|#####2    | 102/193 [00:03<00:02, 34.01it/s]


Repairing epochs:  55%|#####4    | 106/193 [00:03<00:02, 33.44it/s]


Repairing epochs:  57%|#####6    | 110/193 [00:03<00:02, 33.08it/s]


Repairing epochs:  59%|#####9    | 114/193 [00:03<00:02, 34.21it/s]


Repairing epochs:  61%|######1   | 118/193 [00:03<00:02, 33.12it/s]


Repairing epochs:  63%|######3   | 122/193 [00:03<00:02, 31.97it/s]


Repairing epochs:  65%|######5   | 126/193 [00:03<00:02, 32.09it/s]


Repairing epochs:  67%|######7   | 130/193 [00:03<00:01, 32.71it/s]


Repairing epochs:  69%|######9   | 134/193 [00:04<00:01, 33.12it/s]


Repairing epochs:  72%|#######1  | 138/193 [00:04<00:01, 33.40it/s]


Repairing epochs:  74%|#######3  | 142/193 [00:04<00:01, 33.87it/s]


Repairing epochs:  76%|#######5  | 146/193 [00:04<00:01, 33.33it/s]


Repairing epochs:  78%|#######7  | 150/193 [00:04<00:01, 33.68it/s]


Repairing epochs:  80%|#######9  | 154/193 [00:04<00:01, 33.37it/s]


Repairing epochs:  82%|########1 | 158/193 [00:04<00:01, 32.52it/s]


Repairing epochs:  84%|########3 | 162/193 [00:04<00:00, 32.36it/s]


Repairing epochs:  86%|########6 | 166/193 [00:05<00:00, 31.55it/s]


Repairing epochs:  88%|########8 | 170/193 [00:05<00:00, 29.66it/s]


Repairing epochs:  90%|######### | 174/193 [00:05<00:00, 31.00it/s]


Repairing epochs:  92%|#########2| 178/193 [00:05<00:00, 30.44it/s]


Repairing epochs:  94%|#########4| 182/193 [00:05<00:00, 29.94it/s]


Repairing epochs:  96%|#########6| 186/193 [00:05<00:00, 28.20it/s]


Repairing epochs:  98%|#########7| 189/193 [00:05<00:00, 27.63it/s]


Repairing epochs:  99%|#########9| 192/193 [00:05<00:00, 26.95it/s]


Repairing epochs: 100%|##########| 193/193 [00:05<00:00, 32.19it/s]Dropped 55 epochs

We can visualize the cross validation curve over two variables

import numpy as np  # noqa
import matplotlib.pyplot as plt  # noqa
import matplotlib.patches as patches  # noqa
from autoreject import set_matplotlib_defaults  # noqa

set_matplotlib_defaults(plt, style='seaborn-white')
loss = ar.loss_['eeg'].mean(axis=-1)  # losses are stored by channel type.

plt.matshow(loss.T * 1e6, cmap=plt.get_cmap('viridis'))
plt.xticks(range(len(ar.consensus)), ['%.1f' % c for c in ar.consensus])
plt.yticks(range(len(ar.n_interpolate)), ar.n_interpolate)

# Draw rectangle at location of best parameters
ax = plt.gca()
idx, jdx = np.unravel_index(loss.argmin(), loss.shape)
rect = patches.Rectangle((idx - 0.5, jdx - 0.5), 1, 1, linewidth=2,
                         edgecolor='r', facecolor='none')
ax.add_patch(rect)
ax.xaxis.set_ticks_position('bottom')
plt.xlabel(r'Consensus percentage $\kappa$')
plt.ylabel(r'Max sensors interpolated $\rho$')
plt.title('Mean cross validation error (x 1e6)')
plt.colorbar()
plt.show()
../_images/sphx_glr_plot_visualize_bad_epochs_001.png

Out:

/Users/mainak/Documents/github_repos/autoreject/examples/plot_visualize_bad_epochs.py:139: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
  plt.show()

… and visualize the bad epochs and sensors. Bad sensors which have been interpolated are in blue. Bad sensors which are not interpolated are in red. Bad trials are also in red.

scalings = dict(eeg=40e-6)
reject_log.plot_epochs(this_epoch, scalings=scalings)
../_images/sphx_glr_plot_visualize_bad_epochs_002.png

… and the epochs after cleaning with autoreject

epochs_ar.plot(scalings=scalings)
../_images/sphx_glr_plot_visualize_bad_epochs_003.png

The epochs dropped by autoreject are also stored in epochs.drop_log

epochs_ar.plot_drop_log()
../_images/sphx_glr_plot_visualize_bad_epochs_004.png

Finally, the evoked before and after autoreject, for sanity check. We use the spatial_colors argument from MNE as it allows us to see that the eyeblinks have not yet been cleaned but the bad channels have been repaired.

ylim = dict(eeg=(-15, 15))
epochs.average().plot(ylim=ylim, spatial_colors=True)
epochs_ar.average().plot(ylim=ylim, spatial_colors=True)

Total running time of the script: ( 4 minutes 2.261 seconds)

Gallery generated by Sphinx-Gallery

Fork me on GitHub