Note
Go to the end to download the full example code.
Preprocessing workflow with autoreject and ICA#
This example demonstrates how to visualize data when preprocessing
with autoreject and discusses decisions about when and which
other preprocessing steps to use in combination.
tldr: We recommend that you first highpass filter the data, then run autoreject (local) and supply the artifact-free epochs to the ICA algorithm for a robust fit, and finally run autoreject (local) again.
# Author: Alex Rockhill <aprockhill@mailbox.org>
# Mainak Jas <mjas@mgh.harvard.edu>
# Apoorva Karekal <apoorvak@uoregon.edu>
#
# License: BSD-3-Clause
# sphinx_gallery_thumbnail_number = 9
First, we download resting-state EEG data from a Parkinson’s patient
from OpenNeuro. We will do this using openneuro-py which can be
installed with the command pip install openneuro-py.
import os
import matplotlib.pyplot as plt
import openneuro
import mne
import autoreject
dataset = 'ds002778' # The id code on OpenNeuro for this example dataset
subject_id = 'pd14'
target_dir = os.path.join(
os.path.dirname(autoreject.__file__), '..', 'examples', dataset)
os.makedirs(target_dir, exist_ok=True)
openneuro.download(dataset=dataset, target_dir=target_dir,
include=[f'sub-{subject_id}/ses-off'])
👋 Hello! This is openneuro-py 2024.2.0. Great to see you! 🤗
👉 Please report problems 🤯 and bugs 🪲 at
https://github.com/hoechenberger/openneuro-py/issues
🌍 Preparing to download ds002778 …
📁 Traversing directories for ds002778 : 0 entities [00:00, ? entities/s]
📁 Traversing directories for ds002778 : 7 entities [00:00, 29.83 entities/s]
📁 Traversing directories for ds002778 : 10 entities [00:00, 9.36 entities/s]
📁 Traversing directories for ds002778 : 12 entities [00:01, 7.23 entities/s]
📁 Traversing directories for ds002778 : 14 entities [00:01, 6.09 entities/s]
📁 Traversing directories for ds002778 : 15 entities [00:02, 5.87 entities/s]
📁 Traversing directories for ds002778 : 16 entities [00:02, 4.96 entities/s]
📁 Traversing directories for ds002778 : 17 entities [00:02, 4.87 entities/s]
📁 Traversing directories for ds002778 : 18 entities [00:02, 4.54 entities/s]
📁 Traversing directories for ds002778 : 19 entities [00:03, 4.29 entities/s]
📁 Traversing directories for ds002778 : 20 entities [00:03, 4.08 entities/s]
📁 Traversing directories for ds002778 : 21 entities [00:03, 3.91 entities/s]
📁 Traversing directories for ds002778 : 22 entities [00:03, 4.10 entities/s]
📁 Traversing directories for ds002778 : 23 entities [00:04, 4.20 entities/s]
📁 Traversing directories for ds002778 : 24 entities [00:04, 4.99 entities/s]
📁 Traversing directories for ds002778 : 25 entities [00:04, 4.53 entities/s]
📁 Traversing directories for ds002778 : 26 entities [00:04, 5.31 entities/s]
📁 Traversing directories for ds002778 : 27 entities [00:04, 4.85 entities/s]
📁 Traversing directories for ds002778 : 28 entities [00:04, 5.51 entities/s]
📁 Traversing directories for ds002778 : 29 entities [00:05, 5.17 entities/s]
📁 Traversing directories for ds002778 : 30 entities [00:05, 4.48 entities/s]
📁 Traversing directories for ds002778 : 32 entities [00:05, 6.43 entities/s]
📁 Traversing directories for ds002778 : 36 entities [00:05, 10.87 entities/s]
📁 Traversing directories for ds002778 : 38 entities [00:06, 9.65 entities/s]
📁 Traversing directories for ds002778 : 40 entities [00:06, 11.23 entities/s]
📁 Traversing directories for ds002778 : 43 entities [00:06, 11.83 entities/s]
📁 Traversing directories for ds002778 : 45 entities [00:06, 8.30 entities/s]
📁 Traversing directories for ds002778 : 47 entities [00:07, 6.98 entities/s]
📁 Traversing directories for ds002778 : 48 entities [00:07, 7.20 entities/s]
📁 Traversing directories for ds002778 : 49 entities [00:07, 6.60 entities/s]
📁 Traversing directories for ds002778 : 51 entities [00:07, 6.30 entities/s]
📁 Traversing directories for ds002778 : 52 entities [00:08, 6.30 entities/s]
📁 Traversing directories for ds002778 : 53 entities [00:08, 5.64 entities/s]
📁 Traversing directories for ds002778 : 55 entities [00:08, 5.71 entities/s]
📁 Traversing directories for ds002778 : 56 entities [00:08, 6.12 entities/s]
📁 Traversing directories for ds002778 : 57 entities [00:09, 5.26 entities/s]
📁 Traversing directories for ds002778 : 58 entities [00:09, 5.94 entities/s]
📁 Traversing directories for ds002778 : 59 entities [00:09, 5.56 entities/s]
📁 Traversing directories for ds002778 : 60 entities [00:09, 5.95 entities/s]
📁 Traversing directories for ds002778 : 61 entities [00:09, 5.44 entities/s]
📁 Traversing directories for ds002778 : 62 entities [00:09, 5.98 entities/s]
📁 Traversing directories for ds002778 : 63 entities [00:10, 5.67 entities/s]
📁 Traversing directories for ds002778 : 64 entities [00:10, 6.45 entities/s]
📁 Traversing directories for ds002778 : 64 entities [00:10, 6.28 entities/s]
📥 Retrieving up to 12 files (5 concurrent downloads).
Skipping participants.json: already downloaded.: 100%|██████████| 1.24k/1.24k [00:00<?, ?B/s]
Skipping participants.tsv: already downloaded.: 100%|██████████| 1.62k/1.62k [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_scans.tsv: already downloaded.: 100%|██████████| 76.0/76.0 [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_task-rest_beh.tsv: already downloaded.: 100%|██████████| 10.0/10.0 [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_task-rest_channels.tsv: already downloaded.: 100%|██████████| 2.22k/2.22k [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_task-rest_eeg.json: already downloaded.: 100%|██████████| 471/471 [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_task-rest_events.tsv: already downloaded.: 100%|██████████| 65.0/65.0 [00:00<?, ?B/s]
Skipping sub-pd14_ses-off_task-rest_eeg.bdf: already downloaded.: 100%|██████████| 17.5M/17.5M [00:00<?, ?B/s]
Re-downloading sub-pd14_ses-off_task-rest_beh.json: file size mismatch.: 0.00B [00:00, ?B/s]
Re-downloading dataset_description.json: file size mismatch.: 0.00B [00:00, ?B/s]
Re-downloading CHANGES: file size mismatch.: 0.00B [00:00, ?B/s]
Re-downloading README: file size mismatch.: 0.00B [00:00, ?B/s]
✅ Finished downloading ds002778.
🧠 Please enjoy your brains.
We will now load in the raw data from the bdf file downloaded from OpenNeuro and, since this is resting-state data without any events, make regularly spaced events with which to epoch the raw data. In the averaged plot, we can see that there may be some eyeblink artifact contamination but, overall, the data is typical of resting-state EEG.
raw_fname = os.path.join(target_dir, f'sub-{subject_id}', 'ses-off', 'eeg',
'sub-pd14_ses-off_task-rest_eeg.bdf')
raw = mne.io.read_raw_bdf(raw_fname, preload=True)
dig_montage = mne.channels.make_standard_montage('biosemi32')
raw.drop_channels([ch for ch in raw.ch_names
if ch not in dig_montage.ch_names])
raw.set_montage(dig_montage) # use the standard montage
epochs = mne.make_fixed_length_epochs(raw, duration=3, preload=True)
# plot the data
epochs.average().detrend().plot_joint()
Extracting EDF parameters from /home/circleci/project/examples/ds002778/sub-pd14/ses-off/eeg/sub-pd14_ses-off_task-rest_eeg.bdf...
BDF file detected
Setting channel info structure...
Creating raw.info structure...
Reading 0 ... 149503 = 0.000 ... 291.998 secs...
Not setting metadata
97 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 97 events and 1536 original time points ...
0 bad epochs dropped
No projector specified for this dataset. Please consider the method self.add_proj.
<Figure size 800x420 with 6 Axes>
Autoreject without any other preprocessing#
Now, we’ll naively apply autoreject as our first preprocessing step.
As we can see in the plot of the rejected epochs, there are many eyeblinks that caused the epoch to be dropped. This resulted in a lot of the data being lost.
The data looks fairly clean already and we don’t want to interpolate more than a few sensors since we only have 32 to start, so the number of channels to interpolate was set to check some low numbers
ar = autoreject.AutoReject(n_interpolate=[1, 2, 3, 4], random_state=11,
n_jobs=1, verbose=True)
ar.fit(epochs[:20]) # fit on a few epochs to save time
epochs_ar, reject_log = ar.transform(epochs, return_log=True)
Running autoreject on ch_type=eeg
0%| | Creating augmented epochs : 0/32 [00:00<?, ?it/s]
6%|▋ | Creating augmented epochs : 2/32 [00:00<00:00, 73.47it/s]
12%|█▎ | Creating augmented epochs : 4/32 [00:00<00:00, 74.18it/s]
19%|█▉ | Creating augmented epochs : 6/32 [00:00<00:00, 74.96it/s]
25%|██▌ | Creating augmented epochs : 8/32 [00:00<00:00, 76.22it/s]
31%|███▏ | Creating augmented epochs : 10/32 [00:00<00:00, 75.76it/s]
38%|███▊ | Creating augmented epochs : 12/32 [00:00<00:00, 76.49it/s]
44%|████▍ | Creating augmented epochs : 14/32 [00:00<00:00, 76.21it/s]
50%|█████ | Creating augmented epochs : 16/32 [00:00<00:00, 76.01it/s]
56%|█████▋ | Creating augmented epochs : 18/32 [00:00<00:00, 76.36it/s]
62%|██████▎ | Creating augmented epochs : 20/32 [00:00<00:00, 76.12it/s]
69%|██████▉ | Creating augmented epochs : 22/32 [00:00<00:00, 75.83it/s]
75%|███████▌ | Creating augmented epochs : 24/32 [00:00<00:00, 76.11it/s]
81%|████████▏ | Creating augmented epochs : 26/32 [00:00<00:00, 76.34it/s]
88%|████████▊ | Creating augmented epochs : 28/32 [00:00<00:00, 76.20it/s]
94%|█████████▍| Creating augmented epochs : 30/32 [00:00<00:00, 76.13it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 75.87it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 75.86it/s]
0%| | Computing thresholds ... : 0/32 [00:00<?, ?it/s]
3%|▎ | Computing thresholds ... : 1/32 [00:00<00:14, 2.09it/s]
6%|▋ | Computing thresholds ... : 2/32 [00:00<00:14, 2.04it/s]
9%|▉ | Computing thresholds ... : 3/32 [00:01<00:14, 2.02it/s]
12%|█▎ | Computing thresholds ... : 4/32 [00:01<00:13, 2.02it/s]
16%|█▌ | Computing thresholds ... : 5/32 [00:02<00:13, 2.02it/s]
19%|█▉ | Computing thresholds ... : 6/32 [00:02<00:12, 2.02it/s]
22%|██▏ | Computing thresholds ... : 7/32 [00:03<00:12, 2.02it/s]
25%|██▌ | Computing thresholds ... : 8/32 [00:03<00:11, 2.02it/s]
28%|██▊ | Computing thresholds ... : 9/32 [00:04<00:11, 2.02it/s]
31%|███▏ | Computing thresholds ... : 10/32 [00:04<00:10, 2.02it/s]
34%|███▍ | Computing thresholds ... : 11/32 [00:05<00:10, 2.01it/s]
38%|███▊ | Computing thresholds ... : 12/32 [00:05<00:09, 2.01it/s]
41%|████ | Computing thresholds ... : 13/32 [00:06<00:09, 2.01it/s]
44%|████▍ | Computing thresholds ... : 14/32 [00:06<00:08, 2.01it/s]
47%|████▋ | Computing thresholds ... : 15/32 [00:07<00:08, 2.01it/s]
50%|█████ | Computing thresholds ... : 16/32 [00:07<00:07, 2.01it/s]
53%|█████▎ | Computing thresholds ... : 17/32 [00:08<00:07, 2.01it/s]
56%|█████▋ | Computing thresholds ... : 18/32 [00:08<00:06, 2.01it/s]
59%|█████▉ | Computing thresholds ... : 19/32 [00:09<00:06, 2.01it/s]
62%|██████▎ | Computing thresholds ... : 20/32 [00:09<00:05, 2.01it/s]
66%|██████▌ | Computing thresholds ... : 21/32 [00:10<00:05, 2.01it/s]
69%|██████▉ | Computing thresholds ... : 22/32 [00:10<00:04, 2.01it/s]
72%|███████▏ | Computing thresholds ... : 23/32 [00:11<00:04, 2.01it/s]
75%|███████▌ | Computing thresholds ... : 24/32 [00:11<00:03, 2.01it/s]
78%|███████▊ | Computing thresholds ... : 25/32 [00:12<00:03, 2.01it/s]
81%|████████▏ | Computing thresholds ... : 26/32 [00:12<00:02, 2.01it/s]
84%|████████▍ | Computing thresholds ... : 27/32 [00:13<00:02, 2.01it/s]
88%|████████▊ | Computing thresholds ... : 28/32 [00:13<00:01, 2.01it/s]
91%|█████████ | Computing thresholds ... : 29/32 [00:14<00:01, 2.01it/s]
94%|█████████▍| Computing thresholds ... : 30/32 [00:14<00:00, 2.01it/s]
97%|█████████▋| Computing thresholds ... : 31/32 [00:15<00:00, 2.01it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:15<00:00, 2.01it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:15<00:00, 2.01it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 116.98it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 119.95it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 120.50it/s]
40%|████ | Repairing epochs : 8/20 [00:00<00:00, 119.99it/s]
55%|█████▌ | Repairing epochs : 11/20 [00:00<00:00, 126.94it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 135.90it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 141.85it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 143.42it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 140.91it/s]
0%| | n_interp : 0/4 [00:00<?, ?it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 91.57it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 122.92it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 115.93it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 116.81it/s]
65%|██████▌ | Repairing epochs : 13/20 [00:00<00:00, 124.86it/s]
75%|███████▌ | Repairing epochs : 15/20 [00:00<00:00, 118.63it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 114.37it/s]
95%|█████████▌| Repairing epochs : 19/20 [00:00<00:00, 110.38it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 112.29it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 9.60it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 9.83it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 9.83it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 9.94it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 10.01it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 10.11it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 10.18it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 10.26it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 10.07it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.07it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.06it/s]
25%|██▌ | n_interp : 1/4 [00:01<00:03, 1.19s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 102.19it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 134.08it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 119.05it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 118.66it/s]
65%|██████▌ | Repairing epochs : 13/20 [00:00<00:00, 128.56it/s]
75%|███████▌ | Repairing epochs : 15/20 [00:00<00:00, 124.16it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 120.22it/s]
95%|█████████▌| Repairing epochs : 19/20 [00:00<00:00, 116.19it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 119.17it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.51it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 10.82it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 10.98it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 11.07it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 11.16it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 11.19it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 11.13it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 10.95it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 10.93it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.96it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.97it/s]
50%|█████ | n_interp : 2/4 [00:02<00:02, 1.14s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 101.35it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 139.07it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 127.22it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 125.28it/s]
65%|██████▌ | Repairing epochs : 13/20 [00:00<00:00, 135.58it/s]
75%|███████▌ | Repairing epochs : 15/20 [00:00<00:00, 130.48it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 126.38it/s]
95%|█████████▌| Repairing epochs : 19/20 [00:00<00:00, 121.42it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 124.41it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 11.17it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 11.09it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 11.13it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 11.16it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 11.15it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 11.17it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 11.23it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 11.27it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 11.29it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.28it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.26it/s]
75%|███████▌ | n_interp : 3/4 [00:03<00:01, 1.11s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 104.29it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 137.68it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 123.89it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 121.05it/s]
65%|██████▌ | Repairing epochs : 13/20 [00:00<00:00, 131.85it/s]
75%|███████▌ | Repairing epochs : 15/20 [00:00<00:00, 126.80it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 122.24it/s]
95%|█████████▌| Repairing epochs : 19/20 [00:00<00:00, 118.02it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 120.90it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 11.79it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 11.85it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 11.64it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 11.71it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 11.79it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 11.77it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 11.76it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 11.65it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 11.60it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.64it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.65it/s]
100%|██████████| n_interp : 4/4 [00:04<00:00, 1.09s/it]
100%|██████████| n_interp : 4/4 [00:04<00:00, 1.09s/it]
Estimated consensus=0.10 and n_interpolate=1
0%| | Repairing epochs : 0/97 [00:00<?, ?it/s]
1%| | Repairing epochs : 1/97 [00:00<00:02, 35.08it/s]
2%|▏ | Repairing epochs : 2/97 [00:00<00:03, 30.58it/s]
3%|▎ | Repairing epochs : 3/97 [00:00<00:02, 32.05it/s]
4%|▍ | Repairing epochs : 4/97 [00:00<00:02, 32.68it/s]
5%|▌ | Repairing epochs : 5/97 [00:00<00:02, 33.07it/s]
6%|▌ | Repairing epochs : 6/97 [00:00<00:02, 33.38it/s]
7%|▋ | Repairing epochs : 7/97 [00:00<00:02, 32.06it/s]
8%|▊ | Repairing epochs : 8/97 [00:00<00:02, 32.43it/s]
9%|▉ | Repairing epochs : 9/97 [00:00<00:02, 32.64it/s]
10%|█ | Repairing epochs : 10/97 [00:00<00:02, 31.75it/s]
11%|█▏ | Repairing epochs : 11/97 [00:00<00:02, 32.08it/s]
12%|█▏ | Repairing epochs : 12/97 [00:00<00:02, 32.29it/s]
13%|█▎ | Repairing epochs : 13/97 [00:00<00:02, 32.47it/s]
14%|█▍ | Repairing epochs : 14/97 [00:00<00:02, 32.68it/s]
15%|█▌ | Repairing epochs : 15/97 [00:00<00:02, 32.05it/s]
16%|█▋ | Repairing epochs : 16/97 [00:00<00:02, 31.57it/s]
18%|█▊ | Repairing epochs : 17/97 [00:00<00:02, 31.78it/s]
19%|█▊ | Repairing epochs : 18/97 [00:00<00:02, 31.26it/s]
20%|█▉ | Repairing epochs : 19/97 [00:00<00:02, 31.57it/s]
21%|██ | Repairing epochs : 20/97 [00:00<00:02, 31.80it/s]
22%|██▏ | Repairing epochs : 21/97 [00:00<00:02, 32.00it/s]
23%|██▎ | Repairing epochs : 22/97 [00:00<00:02, 32.19it/s]
24%|██▎ | Repairing epochs : 23/97 [00:00<00:02, 32.35it/s]
25%|██▍ | Repairing epochs : 24/97 [00:00<00:02, 31.86it/s]
26%|██▌ | Repairing epochs : 25/97 [00:00<00:02, 32.09it/s]
27%|██▋ | Repairing epochs : 26/97 [00:00<00:02, 32.26it/s]
28%|██▊ | Repairing epochs : 27/97 [00:00<00:02, 31.86it/s]
29%|██▉ | Repairing epochs : 28/97 [00:00<00:02, 31.50it/s]
30%|██▉ | Repairing epochs : 29/97 [00:00<00:02, 31.68it/s]
31%|███ | Repairing epochs : 30/97 [00:00<00:02, 31.78it/s]
32%|███▏ | Repairing epochs : 31/97 [00:00<00:02, 31.40it/s]
33%|███▎ | Repairing epochs : 32/97 [00:01<00:02, 31.59it/s]
34%|███▍ | Repairing epochs : 33/97 [00:01<00:02, 31.73it/s]
35%|███▌ | Repairing epochs : 34/97 [00:01<00:01, 31.87it/s]
36%|███▌ | Repairing epochs : 35/97 [00:01<00:01, 31.56it/s]
37%|███▋ | Repairing epochs : 36/97 [00:01<00:01, 31.71it/s]
38%|███▊ | Repairing epochs : 37/97 [00:01<00:01, 31.39it/s]
39%|███▉ | Repairing epochs : 38/97 [00:01<00:01, 31.58it/s]
40%|████ | Repairing epochs : 39/97 [00:01<00:01, 31.27it/s]
41%|████ | Repairing epochs : 40/97 [00:01<00:01, 31.46it/s]
42%|████▏ | Repairing epochs : 41/97 [00:01<00:01, 31.61it/s]
43%|████▎ | Repairing epochs : 42/97 [00:01<00:01, 31.73it/s]
44%|████▍ | Repairing epochs : 43/97 [00:01<00:01, 31.43it/s]
45%|████▌ | Repairing epochs : 44/97 [00:01<00:01, 31.64it/s]
46%|████▋ | Repairing epochs : 45/97 [00:01<00:01, 31.80it/s]
47%|████▋ | Repairing epochs : 46/97 [00:01<00:01, 31.96it/s]
48%|████▊ | Repairing epochs : 47/97 [00:01<00:01, 31.68it/s]
49%|████▉ | Repairing epochs : 48/97 [00:01<00:01, 31.84it/s]
51%|█████ | Repairing epochs : 49/97 [00:01<00:01, 32.00it/s]
52%|█████▏ | Repairing epochs : 50/97 [00:01<00:01, 32.14it/s]
53%|█████▎ | Repairing epochs : 51/97 [00:01<00:01, 31.85it/s]
54%|█████▎ | Repairing epochs : 52/97 [00:01<00:01, 32.01it/s]
55%|█████▍ | Repairing epochs : 53/97 [00:01<00:01, 32.10it/s]
56%|█████▌ | Repairing epochs : 54/97 [00:01<00:01, 31.74it/s]
57%|█████▋ | Repairing epochs : 55/97 [00:01<00:01, 31.47it/s]
58%|█████▊ | Repairing epochs : 56/97 [00:01<00:01, 31.61it/s]
59%|█████▉ | Repairing epochs : 57/97 [00:01<00:01, 31.72it/s]
60%|█████▉ | Repairing epochs : 58/97 [00:01<00:01, 31.43it/s]
61%|██████ | Repairing epochs : 59/97 [00:01<00:01, 31.16it/s]
62%|██████▏ | Repairing epochs : 60/97 [00:01<00:01, 31.32it/s]
63%|██████▎ | Repairing epochs : 61/97 [00:01<00:01, 31.46it/s]
64%|██████▍ | Repairing epochs : 62/97 [00:01<00:01, 31.58it/s]
65%|██████▍ | Repairing epochs : 63/97 [00:01<00:01, 31.28it/s]
66%|██████▌ | Repairing epochs : 64/97 [00:02<00:01, 31.05it/s]
67%|██████▋ | Repairing epochs : 65/97 [00:02<00:01, 30.83it/s]
68%|██████▊ | Repairing epochs : 66/97 [00:02<00:01, 30.56it/s]
69%|██████▉ | Repairing epochs : 67/97 [00:02<00:00, 30.33it/s]
70%|███████ | Repairing epochs : 68/97 [00:02<00:00, 30.17it/s]
71%|███████ | Repairing epochs : 69/97 [00:02<00:00, 30.03it/s]
72%|███████▏ | Repairing epochs : 70/97 [00:02<00:00, 29.87it/s]
73%|███████▎ | Repairing epochs : 71/97 [00:02<00:00, 29.57it/s]
74%|███████▍ | Repairing epochs : 72/97 [00:02<00:00, 29.44it/s]
75%|███████▌ | Repairing epochs : 73/97 [00:02<00:00, 29.31it/s]
76%|███████▋ | Repairing epochs : 74/97 [00:02<00:00, 29.18it/s]
77%|███████▋ | Repairing epochs : 75/97 [00:02<00:00, 29.08it/s]
78%|███████▊ | Repairing epochs : 76/97 [00:02<00:00, 29.02it/s]
79%|███████▉ | Repairing epochs : 77/97 [00:02<00:00, 28.95it/s]
80%|████████ | Repairing epochs : 78/97 [00:02<00:00, 28.87it/s]
81%|████████▏ | Repairing epochs : 79/97 [00:02<00:00, 28.79it/s]
82%|████████▏ | Repairing epochs : 80/97 [00:02<00:00, 28.73it/s]
84%|████████▎ | Repairing epochs : 81/97 [00:02<00:00, 28.65it/s]
85%|████████▍ | Repairing epochs : 82/97 [00:02<00:00, 28.60it/s]
86%|████████▌ | Repairing epochs : 83/97 [00:02<00:00, 28.55it/s]
87%|████████▋ | Repairing epochs : 84/97 [00:02<00:00, 28.49it/s]
88%|████████▊ | Repairing epochs : 85/97 [00:02<00:00, 28.43it/s]
89%|████████▊ | Repairing epochs : 86/97 [00:02<00:00, 28.36it/s]
90%|████████▉ | Repairing epochs : 87/97 [00:02<00:00, 28.31it/s]
91%|█████████ | Repairing epochs : 88/97 [00:02<00:00, 28.25it/s]
92%|█████████▏| Repairing epochs : 89/97 [00:02<00:00, 28.19it/s]
93%|█████████▎| Repairing epochs : 90/97 [00:02<00:00, 28.13it/s]
94%|█████████▍| Repairing epochs : 91/97 [00:03<00:00, 28.11it/s]
95%|█████████▍| Repairing epochs : 92/97 [00:03<00:00, 28.08it/s]
96%|█████████▌| Repairing epochs : 93/97 [00:03<00:00, 28.07it/s]
97%|█████████▋| Repairing epochs : 94/97 [00:03<00:00, 28.08it/s]
98%|█████████▊| Repairing epochs : 95/97 [00:03<00:00, 28.05it/s]
99%|█████████▉| Repairing epochs : 96/97 [00:03<00:00, 28.02it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 27.98it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 29.95it/s]
Dropped 27 epochs: 54, 58, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 86, 89, 91, 92, 95, 96
visualize the dropped epochs
epochs[reject_log.bad_epochs].plot(scalings=dict(eeg=100e-6))
Using matplotlib as 2D backend.
<MNEBrowseFigure size 800x800 with 4 Axes>
and the reject log
reject_log.plot('horizontal')
<Figure size 1200x600 with 1 Axes>
Autoreject with high-pass filter#
The data may be very valuable and the time for the experiment limited and so we may want to take steps to reduce the number of epochs dropped by first using other steps to preprocess the data. We will use a high-pass filter first to remove slow drift that could cause epochs to be dropped.
When making this decision to filter the data, we do want to be careful because filtering can spread sharp, high-frequency transients and distort the phase of the signal. Most evoked response potential analyses use filtering since the interest is in the time series, but if you are doing a frequency based analysis, filtering before the Fourier transform could potentially be avoided by detrending instead.
raw.filter(l_freq=1, h_freq=None)
epochs = mne.make_fixed_length_epochs(raw, duration=3, preload=True)
ar = autoreject.AutoReject(n_interpolate=[1, 2, 3, 4], random_state=11,
n_jobs=1, verbose=True)
ar.fit(epochs[:20]) # fit on a few epochs to save time
epochs_ar, reject_log = ar.transform(epochs, return_log=True)
Filtering raw data in 1 contiguous segment
Setting up high-pass filter at 1 Hz
FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal highpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 1.00 Hz (-6 dB cutoff frequency: 0.50 Hz)
- Filter length: 1691 samples (3.303 s)
[Parallel(n_jobs=1)]: Done 17 tasks | elapsed: 0.1s
Not setting metadata
97 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 97 events and 1536 original time points ...
0 bad epochs dropped
Running autoreject on ch_type=eeg
0%| | Creating augmented epochs : 0/32 [00:00<?, ?it/s]
6%|▋ | Creating augmented epochs : 2/32 [00:00<00:00, 84.78it/s]
12%|█▎ | Creating augmented epochs : 4/32 [00:00<00:00, 83.59it/s]
19%|█▉ | Creating augmented epochs : 6/32 [00:00<00:00, 83.94it/s]
25%|██▌ | Creating augmented epochs : 8/32 [00:00<00:00, 85.16it/s]
31%|███▏ | Creating augmented epochs : 10/32 [00:00<00:00, 84.29it/s]
38%|███▊ | Creating augmented epochs : 12/32 [00:00<00:00, 84.79it/s]
44%|████▍ | Creating augmented epochs : 14/32 [00:00<00:00, 84.06it/s]
50%|█████ | Creating augmented epochs : 16/32 [00:00<00:00, 83.57it/s]
56%|█████▋ | Creating augmented epochs : 18/32 [00:00<00:00, 84.02it/s]
62%|██████▎ | Creating augmented epochs : 20/32 [00:00<00:00, 83.77it/s]
69%|██████▉ | Creating augmented epochs : 22/32 [00:00<00:00, 83.39it/s]
75%|███████▌ | Creating augmented epochs : 24/32 [00:00<00:00, 83.75it/s]
81%|████████▏ | Creating augmented epochs : 26/32 [00:00<00:00, 83.64it/s]
88%|████████▊ | Creating augmented epochs : 28/32 [00:00<00:00, 83.34it/s]
94%|█████████▍| Creating augmented epochs : 30/32 [00:00<00:00, 83.09it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 82.77it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 82.99it/s]
0%| | Computing thresholds ... : 0/32 [00:00<?, ?it/s]
3%|▎ | Computing thresholds ... : 1/32 [00:00<00:16, 1.91it/s]
6%|▋ | Computing thresholds ... : 2/32 [00:01<00:16, 1.87it/s]
9%|▉ | Computing thresholds ... : 3/32 [00:01<00:15, 1.85it/s]
12%|█▎ | Computing thresholds ... : 4/32 [00:02<00:15, 1.84it/s]
16%|█▌ | Computing thresholds ... : 5/32 [00:02<00:14, 1.84it/s]
19%|█▉ | Computing thresholds ... : 6/32 [00:03<00:14, 1.84it/s]
22%|██▏ | Computing thresholds ... : 7/32 [00:03<00:13, 1.84it/s]
25%|██▌ | Computing thresholds ... : 8/32 [00:04<00:13, 1.84it/s]
28%|██▊ | Computing thresholds ... : 9/32 [00:04<00:12, 1.84it/s]
31%|███▏ | Computing thresholds ... : 10/32 [00:05<00:11, 1.84it/s]
34%|███▍ | Computing thresholds ... : 11/32 [00:05<00:11, 1.84it/s]
38%|███▊ | Computing thresholds ... : 12/32 [00:06<00:10, 1.84it/s]
41%|████ | Computing thresholds ... : 13/32 [00:07<00:10, 1.84it/s]
44%|████▍ | Computing thresholds ... : 14/32 [00:07<00:09, 1.84it/s]
47%|████▋ | Computing thresholds ... : 15/32 [00:08<00:09, 1.84it/s]
50%|█████ | Computing thresholds ... : 16/32 [00:08<00:08, 1.84it/s]
53%|█████▎ | Computing thresholds ... : 17/32 [00:09<00:08, 1.84it/s]
56%|█████▋ | Computing thresholds ... : 18/32 [00:09<00:07, 1.83it/s]
59%|█████▉ | Computing thresholds ... : 19/32 [00:10<00:07, 1.83it/s]
62%|██████▎ | Computing thresholds ... : 20/32 [00:10<00:06, 1.83it/s]
66%|██████▌ | Computing thresholds ... : 21/32 [00:11<00:06, 1.83it/s]
69%|██████▉ | Computing thresholds ... : 22/32 [00:12<00:05, 1.83it/s]
72%|███████▏ | Computing thresholds ... : 23/32 [00:12<00:04, 1.83it/s]
75%|███████▌ | Computing thresholds ... : 24/32 [00:13<00:04, 1.83it/s]
78%|███████▊ | Computing thresholds ... : 25/32 [00:13<00:03, 1.82it/s]
81%|████████▏ | Computing thresholds ... : 26/32 [00:14<00:03, 1.82it/s]
84%|████████▍ | Computing thresholds ... : 27/32 [00:14<00:02, 1.82it/s]
88%|████████▊ | Computing thresholds ... : 28/32 [00:15<00:02, 1.82it/s]
91%|█████████ | Computing thresholds ... : 29/32 [00:15<00:01, 1.82it/s]
94%|█████████▍| Computing thresholds ... : 30/32 [00:16<00:01, 1.82it/s]
97%|█████████▋| Computing thresholds ... : 31/32 [00:16<00:00, 1.82it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:17<00:00, 1.82it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:17<00:00, 1.82it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 109.84it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 109.28it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 112.17it/s]
45%|████▌ | Repairing epochs : 9/20 [00:00<00:00, 122.11it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 127.79it/s]
75%|███████▌ | Repairing epochs : 15/20 [00:00<00:00, 131.05it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 132.33it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 131.81it/s]
0%| | n_interp : 0/4 [00:00<?, ?it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 65.82it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 76.16it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 73.44it/s]
40%|████ | Repairing epochs : 8/20 [00:00<00:00, 76.97it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 80.66it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 82.13it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 83.41it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 80.91it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 78.67it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 76.74it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 77.10it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:01, 8.74it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 9.34it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 9.53it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 9.48it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 9.49it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 9.48it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 9.44it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 9.47it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 9.57it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.68it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.63it/s]
25%|██▌ | n_interp : 1/4 [00:01<00:04, 1.36s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 68.16it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 82.74it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 81.60it/s]
40%|████ | Repairing epochs : 8/20 [00:00<00:00, 85.77it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 91.17it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 93.05it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 94.58it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 90.42it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 89.16it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 87.08it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 87.29it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.16it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 10.06it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 10.08it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 10.04it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 9.67it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 9.61it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 9.57it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 9.54it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 9.52it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.39it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.44it/s]
50%|█████ | n_interp : 2/4 [00:02<00:02, 1.35s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 66.53it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 81.35it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 78.81it/s]
40%|████ | Repairing epochs : 8/20 [00:00<00:00, 82.06it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 85.47it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 86.26it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 87.03it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 83.55it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 81.85it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 80.35it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 80.74it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 9.19it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 9.43it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 9.49it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 9.45it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 9.47it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 9.45it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 9.47it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 9.50it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 9.50it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.44it/s]
100%|██████████| Fold : 10/10 [00:01<00:00, 9.44it/s]
75%|███████▌ | n_interp : 3/4 [00:04<00:01, 1.35s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
10%|█ | Repairing epochs : 2/20 [00:00<00:00, 69.77it/s]
20%|██ | Repairing epochs : 4/20 [00:00<00:00, 81.58it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 77.52it/s]
40%|████ | Repairing epochs : 8/20 [00:00<00:00, 81.26it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 84.14it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 85.06it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 86.48it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 82.75it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 80.12it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 78.85it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 79.37it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.67it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 10.77it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 10.92it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 10.97it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 11.14it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 11.21it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 11.26it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 11.09it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 11.07it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.07it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.07it/s]
100%|██████████| n_interp : 4/4 [00:05<00:00, 1.31s/it]
100%|██████████| n_interp : 4/4 [00:05<00:00, 1.31s/it]
Estimated consensus=0.40 and n_interpolate=4
0%| | Repairing epochs : 0/97 [00:00<?, ?it/s]
1%| | Repairing epochs : 1/97 [00:00<00:03, 27.59it/s]
2%|▏ | Repairing epochs : 2/97 [00:00<00:03, 27.99it/s]
3%|▎ | Repairing epochs : 3/97 [00:00<00:03, 28.15it/s]
4%|▍ | Repairing epochs : 4/97 [00:00<00:03, 29.75it/s]
5%|▌ | Repairing epochs : 5/97 [00:00<00:03, 29.11it/s]
6%|▌ | Repairing epochs : 6/97 [00:00<00:03, 28.82it/s]
7%|▋ | Repairing epochs : 7/97 [00:00<00:03, 28.46it/s]
8%|▊ | Repairing epochs : 8/97 [00:00<00:03, 29.28it/s]
9%|▉ | Repairing epochs : 9/97 [00:00<00:02, 29.88it/s]
10%|█ | Repairing epochs : 10/97 [00:00<00:02, 29.46it/s]
11%|█▏ | Repairing epochs : 11/97 [00:00<00:02, 30.03it/s]
12%|█▏ | Repairing epochs : 12/97 [00:00<00:02, 29.48it/s]
13%|█▎ | Repairing epochs : 13/97 [00:00<00:02, 29.98it/s]
14%|█▍ | Repairing epochs : 14/97 [00:00<00:02, 29.78it/s]
15%|█▌ | Repairing epochs : 15/97 [00:00<00:02, 29.55it/s]
16%|█▋ | Repairing epochs : 16/97 [00:00<00:02, 29.38it/s]
18%|█▊ | Repairing epochs : 17/97 [00:00<00:02, 29.32it/s]
19%|█▊ | Repairing epochs : 18/97 [00:00<00:02, 29.11it/s]
20%|█▉ | Repairing epochs : 19/97 [00:00<00:02, 28.96it/s]
21%|██ | Repairing epochs : 20/97 [00:00<00:02, 28.87it/s]
22%|██▏ | Repairing epochs : 21/97 [00:00<00:02, 28.87it/s]
23%|██▎ | Repairing epochs : 22/97 [00:00<00:02, 28.88it/s]
24%|██▎ | Repairing epochs : 23/97 [00:00<00:02, 28.89it/s]
25%|██▍ | Repairing epochs : 24/97 [00:00<00:02, 28.81it/s]
26%|██▌ | Repairing epochs : 25/97 [00:00<00:02, 28.79it/s]
27%|██▋ | Repairing epochs : 26/97 [00:00<00:02, 28.81it/s]
28%|██▊ | Repairing epochs : 27/97 [00:00<00:02, 28.76it/s]
29%|██▉ | Repairing epochs : 28/97 [00:00<00:02, 28.78it/s]
30%|██▉ | Repairing epochs : 29/97 [00:01<00:02, 28.65it/s]
31%|███ | Repairing epochs : 30/97 [00:01<00:02, 28.55it/s]
32%|███▏ | Repairing epochs : 31/97 [00:01<00:02, 28.47it/s]
33%|███▎ | Repairing epochs : 32/97 [00:01<00:02, 28.45it/s]
34%|███▍ | Repairing epochs : 33/97 [00:01<00:02, 28.40it/s]
35%|███▌ | Repairing epochs : 34/97 [00:01<00:02, 28.37it/s]
36%|███▌ | Repairing epochs : 35/97 [00:01<00:02, 28.38it/s]
37%|███▋ | Repairing epochs : 36/97 [00:01<00:02, 28.33it/s]
38%|███▊ | Repairing epochs : 37/97 [00:01<00:02, 28.30it/s]
39%|███▉ | Repairing epochs : 38/97 [00:01<00:02, 28.32it/s]
40%|████ | Repairing epochs : 39/97 [00:01<00:02, 28.24it/s]
41%|████ | Repairing epochs : 40/97 [00:01<00:02, 28.23it/s]
42%|████▏ | Repairing epochs : 41/97 [00:01<00:01, 28.21it/s]
43%|████▎ | Repairing epochs : 42/97 [00:01<00:01, 28.21it/s]
44%|████▍ | Repairing epochs : 43/97 [00:01<00:01, 28.19it/s]
45%|████▌ | Repairing epochs : 44/97 [00:01<00:01, 28.21it/s]
46%|████▋ | Repairing epochs : 45/97 [00:01<00:01, 28.23it/s]
47%|████▋ | Repairing epochs : 46/97 [00:01<00:01, 28.16it/s]
48%|████▊ | Repairing epochs : 47/97 [00:01<00:01, 28.16it/s]
49%|████▉ | Repairing epochs : 48/97 [00:01<00:01, 28.17it/s]
51%|█████ | Repairing epochs : 49/97 [00:01<00:01, 28.17it/s]
52%|█████▏ | Repairing epochs : 50/97 [00:01<00:01, 28.11it/s]
53%|█████▎ | Repairing epochs : 51/97 [00:01<00:01, 28.10it/s]
54%|█████▎ | Repairing epochs : 52/97 [00:01<00:01, 28.14it/s]
55%|█████▍ | Repairing epochs : 53/97 [00:01<00:01, 28.18it/s]
56%|█████▌ | Repairing epochs : 54/97 [00:01<00:01, 28.08it/s]
57%|█████▋ | Repairing epochs : 55/97 [00:01<00:01, 28.12it/s]
58%|█████▊ | Repairing epochs : 56/97 [00:01<00:01, 28.16it/s]
59%|█████▉ | Repairing epochs : 57/97 [00:02<00:01, 28.16it/s]
60%|█████▉ | Repairing epochs : 58/97 [00:02<00:01, 28.18it/s]
61%|██████ | Repairing epochs : 59/97 [00:02<00:01, 28.20it/s]
62%|██████▏ | Repairing epochs : 60/97 [00:02<00:01, 28.18it/s]
63%|██████▎ | Repairing epochs : 61/97 [00:02<00:01, 28.17it/s]
64%|██████▍ | Repairing epochs : 62/97 [00:02<00:01, 28.22it/s]
65%|██████▍ | Repairing epochs : 63/97 [00:02<00:01, 28.19it/s]
66%|██████▌ | Repairing epochs : 64/97 [00:02<00:01, 28.21it/s]
67%|██████▋ | Repairing epochs : 65/97 [00:02<00:01, 28.15it/s]
68%|██████▊ | Repairing epochs : 66/97 [00:02<00:01, 28.17it/s]
69%|██████▉ | Repairing epochs : 67/97 [00:02<00:01, 28.18it/s]
70%|███████ | Repairing epochs : 68/97 [00:02<00:01, 28.19it/s]
71%|███████ | Repairing epochs : 69/97 [00:02<00:00, 28.20it/s]
72%|███████▏ | Repairing epochs : 70/97 [00:02<00:00, 28.17it/s]
73%|███████▎ | Repairing epochs : 71/97 [00:02<00:00, 28.15it/s]
74%|███████▍ | Repairing epochs : 72/97 [00:02<00:00, 28.13it/s]
75%|███████▌ | Repairing epochs : 73/97 [00:02<00:00, 28.18it/s]
76%|███████▋ | Repairing epochs : 74/97 [00:02<00:00, 28.16it/s]
77%|███████▋ | Repairing epochs : 75/97 [00:02<00:00, 28.19it/s]
78%|███████▊ | Repairing epochs : 76/97 [00:02<00:00, 28.24it/s]
79%|███████▉ | Repairing epochs : 77/97 [00:02<00:00, 28.23it/s]
80%|████████ | Repairing epochs : 78/97 [00:02<00:00, 28.25it/s]
81%|████████▏ | Repairing epochs : 79/97 [00:02<00:00, 28.29it/s]
82%|████████▏ | Repairing epochs : 80/97 [00:02<00:00, 28.33it/s]
84%|████████▎ | Repairing epochs : 81/97 [00:02<00:00, 28.36it/s]
85%|████████▍ | Repairing epochs : 82/97 [00:02<00:00, 28.39it/s]
86%|████████▌ | Repairing epochs : 83/97 [00:02<00:00, 28.40it/s]
87%|████████▋ | Repairing epochs : 84/97 [00:02<00:00, 28.33it/s]
88%|████████▊ | Repairing epochs : 85/97 [00:02<00:00, 28.34it/s]
89%|████████▊ | Repairing epochs : 86/97 [00:03<00:00, 28.36it/s]
90%|████████▉ | Repairing epochs : 87/97 [00:03<00:00, 28.38it/s]
91%|█████████ | Repairing epochs : 88/97 [00:03<00:00, 28.34it/s]
92%|█████████▏| Repairing epochs : 89/97 [00:03<00:00, 28.35it/s]
93%|█████████▎| Repairing epochs : 90/97 [00:03<00:00, 28.38it/s]
94%|█████████▍| Repairing epochs : 91/97 [00:03<00:00, 28.37it/s]
95%|█████████▍| Repairing epochs : 92/97 [00:03<00:00, 28.37it/s]
96%|█████████▌| Repairing epochs : 93/97 [00:03<00:00, 28.40it/s]
97%|█████████▋| Repairing epochs : 94/97 [00:03<00:00, 28.39it/s]
98%|█████████▊| Repairing epochs : 95/97 [00:03<00:00, 28.38it/s]
99%|█████████▉| Repairing epochs : 96/97 [00:03<00:00, 28.39it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 28.42it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 28.40it/s]
Dropped 23 epochs: 62, 63, 64, 65, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 86, 89, 91, 92, 95, 96
visualize the dropped epochs
epochs[reject_log.bad_epochs].plot(scalings=dict(eeg=100e-6))
<MNEBrowseFigure size 800x800 with 4 Axes>
and the reject log. As we can see in the plot, high-pass filtering reduced the number of epochs marked as bad by autoreject. This is because low-frequency drifts can inflate peak-to-peak amplitude when an upward drift aligns with a peak.
reject_log.plot('horizontal')
<Figure size 1200x600 with 1 Axes>
ICA#
Finally, we can apply independent components analysis (ICA) to remove eyeblinks from the data. If our analysis were to be very dependent on sensors at the front of the head, we could skip ICA and use the previous result. However, ICA can increase the amount of usable data by applying a spatial filter that downscales the data in sensors most affected by eyeblink artifacts.
Note that ICA works best if bad segments of the data are removed Hence, we will remove the bad segments from the previous run of autoreject for the benefit of the ICA algorithm.
# compute ICA
ica = mne.preprocessing.ICA(random_state=99)
ica.fit(epochs[~reject_log.bad_epochs])
Fitting ICA to data using 32 channels (please be patient, this may take a while)
Selecting by non-zero PCA components: 32 components
Fitting ICA took 2.9s.
We can see in the plots below that ICA effectively removed eyeblink artifact.
plot source components to see which is made up of blinks
exclude = [0, # blinks
2 # saccades
]
ica.plot_components(exclude)
ica.exclude = exclude
plot with and without eyeblink component
Applying ICA to Evoked instance
Transforming to ICA space (32 components)
Zeroing out 2 ICA components
Projecting back using 32 PCA components
Applying ICA to Epochs instance
Transforming to ICA space (32 components)
Zeroing out 2 ICA components
Projecting back using 32 PCA components
Autoreject with highpass filter and ICA#
We can see in this section that preprocessing, especially ICA, can be made to do a lot of the heavy lifting. There isn’t a huge difference when viewing the averaged data because the ICA effectively limited the number of epochs that had to be dropped. However, there are still artifacts such as non-stereotypical blinks that weren’t able to be removed by ICA, channel “pops” (sharp transients with exponential RC decay), muscle artifact such as jaw clenches and gross movement artifact that could still impact analyses.
These are the basic steps for a workflow with decisions that must be
made based on what the data is being used for. Following this may help
you optimize your use of autoreject in preprocessing.
# compute channel-level rejections
ar = autoreject.AutoReject(n_interpolate=[1, 2, 3, 4], random_state=11,
n_jobs=1, verbose=True)
ar.fit(epochs[:20]) # fit on the first 20 epochs to save time
epochs_ar, reject_log = ar.transform(epochs, return_log=True)
epochs[reject_log.bad_epochs].plot(scalings=dict(eeg=100e-6))
Running autoreject on ch_type=eeg
0%| | Creating augmented epochs : 0/32 [00:00<?, ?it/s]
6%|▋ | Creating augmented epochs : 2/32 [00:00<00:00, 84.36it/s]
12%|█▎ | Creating augmented epochs : 4/32 [00:00<00:00, 83.82it/s]
19%|█▉ | Creating augmented epochs : 6/32 [00:00<00:00, 84.35it/s]
25%|██▌ | Creating augmented epochs : 8/32 [00:00<00:00, 85.79it/s]
31%|███▏ | Creating augmented epochs : 10/32 [00:00<00:00, 85.53it/s]
38%|███▊ | Creating augmented epochs : 12/32 [00:00<00:00, 86.11it/s]
44%|████▍ | Creating augmented epochs : 14/32 [00:00<00:00, 85.85it/s]
50%|█████ | Creating augmented epochs : 16/32 [00:00<00:00, 85.79it/s]
56%|█████▋ | Creating augmented epochs : 18/32 [00:00<00:00, 86.33it/s]
62%|██████▎ | Creating augmented epochs : 20/32 [00:00<00:00, 86.03it/s]
69%|██████▉ | Creating augmented epochs : 22/32 [00:00<00:00, 85.73it/s]
75%|███████▌ | Creating augmented epochs : 24/32 [00:00<00:00, 85.90it/s]
81%|████████▏ | Creating augmented epochs : 26/32 [00:00<00:00, 86.08it/s]
88%|████████▊ | Creating augmented epochs : 28/32 [00:00<00:00, 86.03it/s]
94%|█████████▍| Creating augmented epochs : 30/32 [00:00<00:00, 85.91it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 85.65it/s]
100%|██████████| Creating augmented epochs : 32/32 [00:00<00:00, 85.63it/s]
0%| | Computing thresholds ... : 0/32 [00:00<?, ?it/s]
3%|▎ | Computing thresholds ... : 1/32 [00:00<00:15, 1.94it/s]
6%|▋ | Computing thresholds ... : 2/32 [00:01<00:15, 1.89it/s]
9%|▉ | Computing thresholds ... : 3/32 [00:01<00:15, 1.88it/s]
12%|█▎ | Computing thresholds ... : 4/32 [00:02<00:14, 1.87it/s]
16%|█▌ | Computing thresholds ... : 5/32 [00:02<00:14, 1.87it/s]
19%|█▉ | Computing thresholds ... : 6/32 [00:03<00:13, 1.87it/s]
22%|██▏ | Computing thresholds ... : 7/32 [00:03<00:13, 1.87it/s]
25%|██▌ | Computing thresholds ... : 8/32 [00:04<00:12, 1.87it/s]
28%|██▊ | Computing thresholds ... : 9/32 [00:04<00:12, 1.87it/s]
31%|███▏ | Computing thresholds ... : 10/32 [00:05<00:11, 1.87it/s]
34%|███▍ | Computing thresholds ... : 11/32 [00:05<00:11, 1.87it/s]
38%|███▊ | Computing thresholds ... : 12/32 [00:06<00:10, 1.87it/s]
41%|████ | Computing thresholds ... : 13/32 [00:06<00:10, 1.87it/s]
44%|████▍ | Computing thresholds ... : 14/32 [00:07<00:09, 1.87it/s]
47%|████▋ | Computing thresholds ... : 15/32 [00:08<00:09, 1.87it/s]
50%|█████ | Computing thresholds ... : 16/32 [00:08<00:08, 1.88it/s]
53%|█████▎ | Computing thresholds ... : 17/32 [00:09<00:07, 1.88it/s]
56%|█████▋ | Computing thresholds ... : 18/32 [00:09<00:07, 1.88it/s]
59%|█████▉ | Computing thresholds ... : 19/32 [00:10<00:06, 1.88it/s]
62%|██████▎ | Computing thresholds ... : 20/32 [00:10<00:06, 1.88it/s]
66%|██████▌ | Computing thresholds ... : 21/32 [00:11<00:05, 1.87it/s]
69%|██████▉ | Computing thresholds ... : 22/32 [00:11<00:05, 1.87it/s]
72%|███████▏ | Computing thresholds ... : 23/32 [00:12<00:04, 1.87it/s]
75%|███████▌ | Computing thresholds ... : 24/32 [00:12<00:04, 1.87it/s]
78%|███████▊ | Computing thresholds ... : 25/32 [00:13<00:03, 1.87it/s]
81%|████████▏ | Computing thresholds ... : 26/32 [00:13<00:03, 1.87it/s]
84%|████████▍ | Computing thresholds ... : 27/32 [00:14<00:02, 1.87it/s]
88%|████████▊ | Computing thresholds ... : 28/32 [00:14<00:02, 1.87it/s]
91%|█████████ | Computing thresholds ... : 29/32 [00:15<00:01, 1.87it/s]
94%|█████████▍| Computing thresholds ... : 30/32 [00:16<00:01, 1.86it/s]
97%|█████████▋| Computing thresholds ... : 31/32 [00:16<00:00, 1.86it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:17<00:00, 1.86it/s]
100%|██████████| Computing thresholds ... : 32/32 [00:17<00:00, 1.87it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
15%|█▌ | Repairing epochs : 3/20 [00:00<00:00, 131.24it/s]
30%|███ | Repairing epochs : 6/20 [00:00<00:00, 130.83it/s]
45%|████▌ | Repairing epochs : 9/20 [00:00<00:00, 134.92it/s]
65%|██████▌ | Repairing epochs : 13/20 [00:00<00:00, 150.60it/s]
85%|████████▌ | Repairing epochs : 17/20 [00:00<00:00, 160.03it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 163.13it/s]
0%| | n_interp : 0/4 [00:00<?, ?it/s]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
15%|█▌ | Repairing epochs : 3/20 [00:00<00:00, 184.80it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 146.84it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 136.56it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 136.85it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 132.65it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 129.03it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 117.16it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 109.54it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 103.65it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 107.11it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.77it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 11.03it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 11.05it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 11.12it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 11.17it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 11.08it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 11.04it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 11.10it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 11.09it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.09it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 11.08it/s]
25%|██▌ | n_interp : 1/4 [00:01<00:03, 1.12s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
15%|█▌ | Repairing epochs : 3/20 [00:00<00:00, 181.02it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 135.29it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 123.69it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 124.85it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 122.07it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 117.96it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 109.23it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 104.60it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 100.25it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 102.92it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.38it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 10.21it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 10.27it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 10.33it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 10.40it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 10.25it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 10.25it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 10.26it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 10.31it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.37it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.35it/s]
50%|█████ | n_interp : 2/4 [00:02<00:02, 1.16s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
15%|█▌ | Repairing epochs : 3/20 [00:00<00:00, 172.30it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 132.29it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 126.54it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 125.35it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 122.90it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 120.26it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 110.97it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 104.60it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 99.77it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 102.63it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 10.79it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 10.86it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 10.99it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 10.90it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 10.60it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 10.68it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 10.63it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 10.64it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 10.61it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.66it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 10.66it/s]
75%|███████▌ | n_interp : 3/4 [00:03<00:01, 1.16s/it]
0%| | Repairing epochs : 0/20 [00:00<?, ?it/s]
15%|█▌ | Repairing epochs : 3/20 [00:00<00:00, 184.70it/s]
25%|██▌ | Repairing epochs : 5/20 [00:00<00:00, 138.18it/s]
35%|███▌ | Repairing epochs : 7/20 [00:00<00:00, 129.20it/s]
50%|█████ | Repairing epochs : 10/20 [00:00<00:00, 128.60it/s]
60%|██████ | Repairing epochs : 12/20 [00:00<00:00, 124.83it/s]
70%|███████ | Repairing epochs : 14/20 [00:00<00:00, 122.12it/s]
80%|████████ | Repairing epochs : 16/20 [00:00<00:00, 112.01it/s]
90%|█████████ | Repairing epochs : 18/20 [00:00<00:00, 106.17it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 101.54it/s]
100%|██████████| Repairing epochs : 20/20 [00:00<00:00, 104.47it/s]
0%| | Fold : 0/10 [00:00<?, ?it/s]
10%|█ | Fold : 1/10 [00:00<00:00, 12.44it/s]
20%|██ | Fold : 2/10 [00:00<00:00, 12.49it/s]
30%|███ | Fold : 3/10 [00:00<00:00, 12.46it/s]
40%|████ | Fold : 4/10 [00:00<00:00, 12.38it/s]
50%|█████ | Fold : 5/10 [00:00<00:00, 12.34it/s]
60%|██████ | Fold : 6/10 [00:00<00:00, 12.27it/s]
70%|███████ | Fold : 7/10 [00:00<00:00, 12.20it/s]
80%|████████ | Fold : 8/10 [00:00<00:00, 12.29it/s]
90%|█████████ | Fold : 9/10 [00:00<00:00, 12.17it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 12.15it/s]
100%|██████████| Fold : 10/10 [00:00<00:00, 12.17it/s]
100%|██████████| n_interp : 4/4 [00:04<00:00, 1.12s/it]
100%|██████████| n_interp : 4/4 [00:04<00:00, 1.13s/it]
Estimated consensus=0.40 and n_interpolate=2
0%| | Repairing epochs : 0/97 [00:00<?, ?it/s]
1%| | Repairing epochs : 1/97 [00:00<00:02, 35.26it/s]
2%|▏ | Repairing epochs : 2/97 [00:00<00:02, 35.13it/s]
3%|▎ | Repairing epochs : 3/97 [00:00<00:02, 34.99it/s]
4%|▍ | Repairing epochs : 4/97 [00:00<00:02, 35.07it/s]
5%|▌ | Repairing epochs : 5/97 [00:00<00:02, 32.96it/s]
6%|▌ | Repairing epochs : 6/97 [00:00<00:02, 32.05it/s]
7%|▋ | Repairing epochs : 7/97 [00:00<00:02, 32.64it/s]
8%|▊ | Repairing epochs : 8/97 [00:00<00:02, 33.06it/s]
9%|▉ | Repairing epochs : 9/97 [00:00<00:02, 33.31it/s]
10%|█ | Repairing epochs : 10/97 [00:00<00:02, 32.55it/s]
11%|█▏ | Repairing epochs : 11/97 [00:00<00:02, 32.94it/s]
12%|█▏ | Repairing epochs : 12/97 [00:00<00:02, 32.32it/s]
13%|█▎ | Repairing epochs : 13/97 [00:00<00:02, 32.69it/s]
14%|█▍ | Repairing epochs : 14/97 [00:00<00:02, 32.12it/s]
15%|█▌ | Repairing epochs : 15/97 [00:00<00:02, 31.65it/s]
16%|█▋ | Repairing epochs : 16/97 [00:00<00:02, 31.39it/s]
18%|█▊ | Repairing epochs : 17/97 [00:00<00:02, 31.12it/s]
19%|█▊ | Repairing epochs : 18/97 [00:00<00:02, 30.87it/s]
20%|█▉ | Repairing epochs : 19/97 [00:00<00:02, 30.67it/s]
21%|██ | Repairing epochs : 20/97 [00:00<00:02, 30.36it/s]
22%|██▏ | Repairing epochs : 21/97 [00:00<00:02, 30.13it/s]
23%|██▎ | Repairing epochs : 22/97 [00:00<00:02, 29.98it/s]
24%|██▎ | Repairing epochs : 23/97 [00:00<00:02, 29.92it/s]
25%|██▍ | Repairing epochs : 24/97 [00:00<00:02, 29.84it/s]
26%|██▌ | Repairing epochs : 25/97 [00:00<00:02, 29.74it/s]
27%|██▋ | Repairing epochs : 26/97 [00:00<00:02, 29.72it/s]
28%|██▊ | Repairing epochs : 27/97 [00:00<00:02, 29.55it/s]
29%|██▉ | Repairing epochs : 28/97 [00:00<00:02, 29.38it/s]
30%|██▉ | Repairing epochs : 29/97 [00:00<00:02, 29.35it/s]
31%|███ | Repairing epochs : 30/97 [00:00<00:02, 29.25it/s]
32%|███▏ | Repairing epochs : 31/97 [00:01<00:02, 29.09it/s]
33%|███▎ | Repairing epochs : 32/97 [00:01<00:02, 28.97it/s]
34%|███▍ | Repairing epochs : 33/97 [00:01<00:02, 28.84it/s]
35%|███▌ | Repairing epochs : 34/97 [00:01<00:02, 28.80it/s]
36%|███▌ | Repairing epochs : 35/97 [00:01<00:02, 28.73it/s]
37%|███▋ | Repairing epochs : 36/97 [00:01<00:02, 28.64it/s]
38%|███▊ | Repairing epochs : 37/97 [00:01<00:02, 28.58it/s]
39%|███▉ | Repairing epochs : 38/97 [00:01<00:02, 28.49it/s]
40%|████ | Repairing epochs : 39/97 [00:01<00:02, 28.40it/s]
41%|████ | Repairing epochs : 40/97 [00:01<00:01, 28.71it/s]
42%|████▏ | Repairing epochs : 41/97 [00:01<00:01, 28.63it/s]
43%|████▎ | Repairing epochs : 42/97 [00:01<00:01, 28.59it/s]
44%|████▍ | Repairing epochs : 43/97 [00:01<00:01, 28.54it/s]
45%|████▌ | Repairing epochs : 44/97 [00:01<00:01, 28.50it/s]
46%|████▋ | Repairing epochs : 45/97 [00:01<00:01, 28.49it/s]
47%|████▋ | Repairing epochs : 46/97 [00:01<00:01, 28.44it/s]
48%|████▊ | Repairing epochs : 47/97 [00:01<00:01, 28.46it/s]
49%|████▉ | Repairing epochs : 48/97 [00:01<00:01, 28.39it/s]
51%|█████ | Repairing epochs : 49/97 [00:01<00:01, 28.71it/s]
52%|█████▏ | Repairing epochs : 50/97 [00:01<00:01, 28.62it/s]
53%|█████▎ | Repairing epochs : 51/97 [00:01<00:01, 28.58it/s]
54%|█████▎ | Repairing epochs : 52/97 [00:01<00:01, 28.59it/s]
55%|█████▍ | Repairing epochs : 53/97 [00:01<00:01, 28.54it/s]
56%|█████▌ | Repairing epochs : 54/97 [00:01<00:01, 28.49it/s]
57%|█████▋ | Repairing epochs : 55/97 [00:01<00:01, 28.50it/s]
58%|█████▊ | Repairing epochs : 56/97 [00:01<00:01, 28.46it/s]
59%|█████▉ | Repairing epochs : 57/97 [00:01<00:01, 28.74it/s]
60%|█████▉ | Repairing epochs : 58/97 [00:01<00:01, 28.66it/s]
61%|██████ | Repairing epochs : 59/97 [00:02<00:01, 28.94it/s]
62%|██████▏ | Repairing epochs : 60/97 [00:02<00:01, 28.85it/s]
63%|██████▎ | Repairing epochs : 61/97 [00:02<00:01, 28.78it/s]
64%|██████▍ | Repairing epochs : 62/97 [00:02<00:01, 29.07it/s]
65%|██████▍ | Repairing epochs : 63/97 [00:02<00:01, 29.00it/s]
66%|██████▌ | Repairing epochs : 64/97 [00:02<00:01, 28.94it/s]
67%|██████▋ | Repairing epochs : 65/97 [00:02<00:01, 28.90it/s]
68%|██████▊ | Repairing epochs : 66/97 [00:02<00:01, 28.85it/s]
69%|██████▉ | Repairing epochs : 67/97 [00:02<00:01, 28.76it/s]
70%|███████ | Repairing epochs : 68/97 [00:02<00:01, 28.69it/s]
71%|███████ | Repairing epochs : 69/97 [00:02<00:00, 28.58it/s]
72%|███████▏ | Repairing epochs : 70/97 [00:02<00:00, 28.53it/s]
73%|███████▎ | Repairing epochs : 71/97 [00:02<00:00, 28.53it/s]
74%|███████▍ | Repairing epochs : 72/97 [00:02<00:00, 28.50it/s]
75%|███████▌ | Repairing epochs : 73/97 [00:02<00:00, 28.50it/s]
76%|███████▋ | Repairing epochs : 74/97 [00:02<00:00, 28.45it/s]
77%|███████▋ | Repairing epochs : 75/97 [00:02<00:00, 28.39it/s]
78%|███████▊ | Repairing epochs : 76/97 [00:02<00:00, 28.40it/s]
79%|███████▉ | Repairing epochs : 77/97 [00:02<00:00, 28.40it/s]
80%|████████ | Repairing epochs : 78/97 [00:02<00:00, 28.34it/s]
81%|████████▏ | Repairing epochs : 79/97 [00:02<00:00, 28.31it/s]
82%|████████▏ | Repairing epochs : 80/97 [00:02<00:00, 28.26it/s]
84%|████████▎ | Repairing epochs : 81/97 [00:02<00:00, 28.18it/s]
85%|████████▍ | Repairing epochs : 82/97 [00:02<00:00, 28.15it/s]
86%|████████▌ | Repairing epochs : 83/97 [00:02<00:00, 28.13it/s]
87%|████████▋ | Repairing epochs : 84/97 [00:02<00:00, 28.09it/s]
88%|████████▊ | Repairing epochs : 85/97 [00:02<00:00, 28.07it/s]
89%|████████▊ | Repairing epochs : 86/97 [00:02<00:00, 28.08it/s]
90%|████████▉ | Repairing epochs : 87/97 [00:03<00:00, 28.05it/s]
91%|█████████ | Repairing epochs : 88/97 [00:03<00:00, 28.02it/s]
92%|█████████▏| Repairing epochs : 89/97 [00:03<00:00, 27.99it/s]
93%|█████████▎| Repairing epochs : 90/97 [00:03<00:00, 27.93it/s]
94%|█████████▍| Repairing epochs : 91/97 [00:03<00:00, 27.93it/s]
95%|█████████▍| Repairing epochs : 92/97 [00:03<00:00, 27.88it/s]
96%|█████████▌| Repairing epochs : 93/97 [00:03<00:00, 27.90it/s]
97%|█████████▋| Repairing epochs : 94/97 [00:03<00:00, 27.91it/s]
98%|█████████▊| Repairing epochs : 95/97 [00:03<00:00, 27.92it/s]
99%|█████████▉| Repairing epochs : 96/97 [00:03<00:00, 27.88it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 27.84it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 28.67it/s]
Dropped 11 epochs: 62, 63, 64, 65, 69, 71, 72, 73, 80, 95, 96
<MNEBrowseFigure size 800x800 with 4 Axes>
We will do a few more visualizations to see that removing the bad epochs
found by autoreject is still important even with preprocessing first.
This is especially important if your analyses include trial-level statistics
such as looking for bursting activity. We’ll visualize why autoreject
excluded these epochs and the effect that including these bad epochs would
have on the data.
First, we will visualize the reject log
reject_log.plot('horizontal')
<Figure size 1200x600 with 1 Axes>
Next, we will visualize the cleaned average data and compare it against the bad segments.
evoked_bad = epochs[reject_log.bad_epochs].average()
plt.figure()
plt.plot(evoked_bad.times, evoked_bad.data.T * 1e6, 'r', zorder=-1)
epochs_ar.average().plot(axes=plt.gca())
<Figure size 640x480 with 2 Axes>
As a last optional step, we can do inspect the reject_log and make manual corrections to the reject_log. For instance, if data is limited, we may not want to drop epochs but retain the list of bad epochs for quality assurance metrics.
reject_log = ar.get_reject_log(epochs)
bad_epochs = reject_log.bad_epochs.copy()
reject_log.bad_epochs[:] = False # no bad epochs
The modified reject log can be applied to the data as follows.
epochs_ar = ar.transform(epochs, reject_log=reject_log)
print(f'Number of epochs originally: {len(epochs)}, '
f'after autoreject: {len(epochs_ar)}')
0%| | Repairing epochs : 0/97 [00:00<?, ?it/s]
1%| | Repairing epochs : 1/97 [00:00<00:02, 34.68it/s]
2%|▏ | Repairing epochs : 2/97 [00:00<00:02, 34.64it/s]
3%|▎ | Repairing epochs : 3/97 [00:00<00:02, 34.62it/s]
4%|▍ | Repairing epochs : 4/97 [00:00<00:02, 34.64it/s]
5%|▌ | Repairing epochs : 5/97 [00:00<00:02, 32.64it/s]
6%|▌ | Repairing epochs : 6/97 [00:00<00:02, 31.73it/s]
7%|▋ | Repairing epochs : 7/97 [00:00<00:02, 32.32it/s]
8%|▊ | Repairing epochs : 8/97 [00:00<00:02, 32.69it/s]
9%|▉ | Repairing epochs : 9/97 [00:00<00:02, 33.01it/s]
10%|█ | Repairing epochs : 10/97 [00:00<00:02, 32.26it/s]
11%|█▏ | Repairing epochs : 11/97 [00:00<00:02, 32.63it/s]
12%|█▏ | Repairing epochs : 12/97 [00:00<00:02, 31.87it/s]
13%|█▎ | Repairing epochs : 13/97 [00:00<00:02, 32.20it/s]
14%|█▍ | Repairing epochs : 14/97 [00:00<00:02, 31.65it/s]
15%|█▌ | Repairing epochs : 15/97 [00:00<00:02, 31.24it/s]
16%|█▋ | Repairing epochs : 16/97 [00:00<00:02, 30.96it/s]
18%|█▊ | Repairing epochs : 17/97 [00:00<00:02, 30.66it/s]
19%|█▊ | Repairing epochs : 18/97 [00:00<00:02, 30.37it/s]
20%|█▉ | Repairing epochs : 19/97 [00:00<00:02, 30.20it/s]
21%|██ | Repairing epochs : 20/97 [00:00<00:02, 30.00it/s]
22%|██▏ | Repairing epochs : 21/97 [00:00<00:02, 29.91it/s]
23%|██▎ | Repairing epochs : 22/97 [00:00<00:02, 29.73it/s]
24%|██▎ | Repairing epochs : 23/97 [00:00<00:02, 29.65it/s]
25%|██▍ | Repairing epochs : 24/97 [00:00<00:02, 29.49it/s]
26%|██▌ | Repairing epochs : 25/97 [00:00<00:02, 29.35it/s]
27%|██▋ | Repairing epochs : 26/97 [00:00<00:02, 29.27it/s]
28%|██▊ | Repairing epochs : 27/97 [00:00<00:02, 29.11it/s]
29%|██▉ | Repairing epochs : 28/97 [00:00<00:02, 28.92it/s]
30%|██▉ | Repairing epochs : 29/97 [00:00<00:02, 28.89it/s]
31%|███ | Repairing epochs : 30/97 [00:01<00:02, 28.83it/s]
32%|███▏ | Repairing epochs : 31/97 [00:01<00:02, 28.70it/s]
33%|███▎ | Repairing epochs : 32/97 [00:01<00:02, 28.65it/s]
34%|███▍ | Repairing epochs : 33/97 [00:01<00:02, 28.69it/s]
35%|███▌ | Repairing epochs : 34/97 [00:01<00:02, 28.71it/s]
36%|███▌ | Repairing epochs : 35/97 [00:01<00:02, 28.70it/s]
37%|███▋ | Repairing epochs : 36/97 [00:01<00:02, 28.66it/s]
38%|███▊ | Repairing epochs : 37/97 [00:01<00:02, 28.65it/s]
39%|███▉ | Repairing epochs : 38/97 [00:01<00:02, 28.59it/s]
40%|████ | Repairing epochs : 39/97 [00:01<00:02, 28.56it/s]
41%|████ | Repairing epochs : 40/97 [00:01<00:01, 28.87it/s]
42%|████▏ | Repairing epochs : 41/97 [00:01<00:01, 28.79it/s]
43%|████▎ | Repairing epochs : 42/97 [00:01<00:01, 28.73it/s]
44%|████▍ | Repairing epochs : 43/97 [00:01<00:01, 28.70it/s]
45%|████▌ | Repairing epochs : 44/97 [00:01<00:01, 28.67it/s]
46%|████▋ | Repairing epochs : 45/97 [00:01<00:01, 28.70it/s]
47%|████▋ | Repairing epochs : 46/97 [00:01<00:01, 28.68it/s]
48%|████▊ | Repairing epochs : 47/97 [00:01<00:01, 28.72it/s]
49%|████▉ | Repairing epochs : 48/97 [00:01<00:01, 28.66it/s]
51%|█████ | Repairing epochs : 49/97 [00:01<00:01, 28.98it/s]
52%|█████▏ | Repairing epochs : 50/97 [00:01<00:01, 28.88it/s]
53%|█████▎ | Repairing epochs : 51/97 [00:01<00:01, 28.87it/s]
54%|█████▎ | Repairing epochs : 52/97 [00:01<00:01, 28.88it/s]
55%|█████▍ | Repairing epochs : 53/97 [00:01<00:01, 28.82it/s]
56%|█████▌ | Repairing epochs : 54/97 [00:01<00:01, 28.72it/s]
57%|█████▋ | Repairing epochs : 55/97 [00:01<00:01, 28.71it/s]
58%|█████▊ | Repairing epochs : 56/97 [00:01<00:01, 28.65it/s]
59%|█████▉ | Repairing epochs : 57/97 [00:01<00:01, 28.92it/s]
60%|█████▉ | Repairing epochs : 58/97 [00:01<00:01, 28.81it/s]
61%|██████ | Repairing epochs : 59/97 [00:02<00:01, 29.08it/s]
62%|██████▏ | Repairing epochs : 60/97 [00:02<00:01, 28.97it/s]
63%|██████▎ | Repairing epochs : 61/97 [00:02<00:01, 28.90it/s]
64%|██████▍ | Repairing epochs : 62/97 [00:02<00:01, 29.18it/s]
65%|██████▍ | Repairing epochs : 63/97 [00:02<00:01, 29.12it/s]
66%|██████▌ | Repairing epochs : 64/97 [00:02<00:01, 29.05it/s]
67%|██████▋ | Repairing epochs : 65/97 [00:02<00:01, 29.01it/s]
68%|██████▊ | Repairing epochs : 66/97 [00:02<00:01, 28.90it/s]
69%|██████▉ | Repairing epochs : 67/97 [00:02<00:01, 28.84it/s]
70%|███████ | Repairing epochs : 68/97 [00:02<00:01, 28.78it/s]
71%|███████ | Repairing epochs : 69/97 [00:02<00:00, 28.72it/s]
72%|███████▏ | Repairing epochs : 70/97 [00:02<00:00, 28.69it/s]
73%|███████▎ | Repairing epochs : 71/97 [00:02<00:00, 28.71it/s]
74%|███████▍ | Repairing epochs : 72/97 [00:02<00:00, 28.66it/s]
75%|███████▌ | Repairing epochs : 73/97 [00:02<00:00, 28.65it/s]
76%|███████▋ | Repairing epochs : 74/97 [00:02<00:00, 28.56it/s]
77%|███████▋ | Repairing epochs : 75/97 [00:02<00:00, 28.52it/s]
78%|███████▊ | Repairing epochs : 76/97 [00:02<00:00, 28.53it/s]
79%|███████▉ | Repairing epochs : 77/97 [00:02<00:00, 28.53it/s]
80%|████████ | Repairing epochs : 78/97 [00:02<00:00, 28.46it/s]
81%|████████▏ | Repairing epochs : 79/97 [00:02<00:00, 28.43it/s]
82%|████████▏ | Repairing epochs : 80/97 [00:02<00:00, 28.37it/s]
84%|████████▎ | Repairing epochs : 81/97 [00:02<00:00, 28.33it/s]
85%|████████▍ | Repairing epochs : 82/97 [00:02<00:00, 28.33it/s]
86%|████████▌ | Repairing epochs : 83/97 [00:02<00:00, 28.30it/s]
87%|████████▋ | Repairing epochs : 84/97 [00:02<00:00, 28.25it/s]
88%|████████▊ | Repairing epochs : 85/97 [00:02<00:00, 28.23it/s]
89%|████████▊ | Repairing epochs : 86/97 [00:02<00:00, 28.23it/s]
90%|████████▉ | Repairing epochs : 87/97 [00:03<00:00, 28.21it/s]
91%|█████████ | Repairing epochs : 88/97 [00:03<00:00, 28.18it/s]
92%|█████████▏| Repairing epochs : 89/97 [00:03<00:00, 28.16it/s]
93%|█████████▎| Repairing epochs : 90/97 [00:03<00:00, 28.08it/s]
94%|█████████▍| Repairing epochs : 91/97 [00:03<00:00, 28.07it/s]
95%|█████████▍| Repairing epochs : 92/97 [00:03<00:00, 27.97it/s]
96%|█████████▌| Repairing epochs : 93/97 [00:03<00:00, 27.96it/s]
97%|█████████▋| Repairing epochs : 94/97 [00:03<00:00, 27.94it/s]
98%|█████████▊| Repairing epochs : 95/97 [00:03<00:00, 27.95it/s]
99%|█████████▉| Repairing epochs : 96/97 [00:03<00:00, 27.89it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 27.86it/s]
100%|██████████| Repairing epochs : 97/97 [00:03<00:00, 28.70it/s]
No bad epochs were found for your data. Returning a copy of the data you wanted to clean. Interpolation may have been done.
Number of epochs originally: 97, after autoreject: 97
Finally, don’t forget that we are working with resting state data here. Here we used long epochs of 3 seconds so that frequency-domain analysis was possible with the epochs. However, this could also lead to longer segments of the data being rejected. If you want more fine-grained control over the artifacts, you can construct shorter epochs and use the autoreject log to mark annotations in MNE that can be used to reject the data during doing time-frequency analysis. We want to emphasize that there is no subsitute for visual inspection. Irrespective of the rejection method used, we highly recommend users to inspect their preprocessed data before further analyses.
Total running time of the script: (1 minutes 47.120 seconds)