Meg analysis pipeline - Revision history

Jstout at 16:04, 21 August 2020

2020-08-21T16:04:41Z

← Older revision		Revision as of 12:04, 21 August 2020
Line 44:		Line 44:
	===Nipype===		===Nipype===
	[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox] </br></br>		[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox] </br></br>
	Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.		Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment. </br>

	[[File:Pipeline_variables.png \| ~~left~~ \| 400px]]		[[File:Pipeline_variables.png \| center \| 400px]] </br>

		⚫	[[File:Nipype pipeline.png\| center \| 400px]] </br>

⚫	[[File:Nipype pipeline.png\| ~~right~~ \| 400px]]

	</br></br>

	===Git===		===Git===

Jstout: /* Nipype */

2020-08-21T15:58:14Z

Nipype

← Older revision		Revision as of 11:58, 21 August 2020
Line 49:		Line 49:

	[[File:Nipype pipeline.png\| right \| 400px]]		[[File:Nipype pipeline.png\| right \| 400px]]

			</br></br>

	===Git===		===Git===

Jstout: /* Nipype */

2020-08-21T15:57:49Z

Nipype

← Older revision		Revision as of 11:57, 21 August 2020
Line 46:		Line 46:
	Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.		Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.

	[[File:Pipeline_variables.png \| ~~center~~ \| 400px]]		[[File:Pipeline_variables.png \| left \| 400px]]
	</br></br>

	[[File:Nipype pipeline.png\|~~center~~ \| 400px]]		[[File:Nipype pipeline.png\| right \| 400px]]

	===Git===		===Git===

Jstout: /* Nipype */

2020-08-21T15:56:46Z

Nipype

← Older revision		Revision as of 11:56, 21 August 2020
Line 43:		Line 43:

	===Nipype===		===Nipype===
	[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]		[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox] </br></br>
	Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.		Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.

Jstout: /* Nipype */

2020-08-21T15:55:21Z

Nipype

← Older revision		Revision as of 11:55, 21 August 2020
Line 45:		Line 45:
	[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]		[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]
	Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.		Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.

			[[File:Pipeline_variables.png \| center \| 400px]]
			</br></br>

	[[File:Nipype pipeline.png\|center \| 400px]]		[[File:Nipype pipeline.png\|center \| 400px]]

Jstout at 15:49, 21 August 2020

2020-08-21T15:49:46Z

← Older revision		Revision as of 11:49, 21 August 2020
Line 1:		Line 1:

			!!! UNDER CONSTRUCTION !!!


	====Reproducible MEG Analysis====		====Reproducible MEG Analysis====
	MEG data collection and analysis requires a lot of interacting components (Stimulus delivery, logfiles, projector onset triggers, analog and digital triggering, electrophysiology measures, MRI coregistration...) that can lead to heterogeneous data and data quality. This makes pipeline analysis particularly difficult due to the variability in inputs. Additionally, with the multitude of techniques and metrics to evaluate MEG data (dipole/MNE/dSPM/beamformer/DICS beamformer/parametric stats/non-parametric stats ....) projects can have evolving analysis protocols. Knowing what preprocessing was performed on what datasets can often come down to an excel spreadsheet or text document. Since these projects can sometimes last multiple years, it is important to know what was done to produce a specific result and how to apply these analysis techniques to new incoming data.		MEG data collection and analysis requires a lot of interacting components (Stimulus delivery, logfiles, projector onset triggers, analog and digital triggering, electrophysiology measures, MRI coregistration...) that can lead to heterogeneous data and data quality. This makes pipeline analysis particularly difficult due to the variability in inputs. Additionally, with the multitude of techniques and metrics to evaluate MEG data (dipole/MNE/dSPM/beamformer/DICS beamformer/parametric stats/non-parametric stats ....) projects can have evolving analysis protocols. Knowing what preprocessing was performed on what datasets can often come down to an excel spreadsheet or text document. Since these projects can sometimes last multiple years, it is important to know what was done to produce a specific result and how to apply these analysis techniques to new incoming data.

Jstout at 15:48, 21 August 2020

2020-08-21T15:48:45Z

← Older revision		Revision as of 11:48, 21 August 2020
Line 1:		Line 1:
	====Reproducible MEG Analysis====		====Reproducible MEG Analysis====
	MEG data collection and analysis requires a lot of interacting components (Stimulus delivery, logfiles, projector onset triggers, analog and digital triggering, electrophysiology measures, MRI coregistration...) that can lead to heterogeneous data and data quality. This makes pipeline analysis particularly difficult due to the variability in inputs. Additionally, with the multitude of techniques and metrics to evaluate MEG data (dipole/MNE/dSPM/beamformer/DICS beamformer/parametric stats/non-parametric stats ....) projects can have evolving analysis protocols. Knowing what preprocessing was performed on what datasets can often come down to an excel spreadsheet or text document. Since these projects can sometimes last multiple years, it is important to know what was done to produce a specific result and how to apply these analysis techniques to new incoming data.		MEG data collection and analysis requires a lot of interacting components (Stimulus delivery, logfiles, projector onset triggers, analog and digital triggering, electrophysiology measures, MRI coregistration...) that can lead to heterogeneous data and data quality. This makes pipeline analysis particularly difficult due to the variability in inputs. Additionally, with the multitude of techniques and metrics to evaluate MEG data (dipole/MNE/dSPM/beamformer/DICS beamformer/parametric stats/non-parametric stats ....) projects can have evolving analysis protocols. Knowing what preprocessing was performed on what datasets can often come down to an excel spreadsheet or text document. Since these projects can sometimes last multiple years, it is important to know what was done to produce a specific result and how to apply these analysis techniques to new incoming data.

			===Check Inputs===
			#Verify that inputs are present
			#Find raw MRI, raw MEG, logfiles
			#Test example below - this is automated if MRIpath/MEGpath/LogfilePath set
			num_mri = len(glob.glob(os.path.join(mri_path, subjid))
			print('{} has {} MRIs present'.format(subjid, num_mri))

			===QA datasets===
			#Create QA scripts for each task
			assert len(2back) == 40
			assert len(correct_response) > 20

	===Containerized Computing Environment===		===Containerized Computing Environment===
Line 28:		Line 40:
	===Nipype===		===Nipype===
	[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]		[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]
			Nipype uses compute modules that can stitch together Bash, Python, and Matlab calls. There are also modules available for all major MRI/fMRI software packages (afni, spm, freesurfer, fsl ...). These modules produce a defined input/output requirements. For example a pipeline will fail all dependency processes if a required output file is not generated. We have developed nipype modules for the major computing components of our data analysis pipeline and have some general analysis pipelines available for the NIMH data (see image below). Nipype additionally documents all output provenance at runtime - including shell variables and computing environment.

	[[File:Nipype pipeline.png\|center \| 400px]]		[[File:Nipype pipeline.png\|center \| 400px]]

Jstout: /* Containerized Computing Environment */

2020-08-21T15:34:50Z

Containerized Computing Environment

← Older revision		Revision as of 11:34, 21 August 2020
Line 5:		Line 5:
	[https://sylabs.io/docs/ Singularity Containers]		[https://sylabs.io/docs/ Singularity Containers]

			#It can be easy to build a container from a current conda environment file
			conda env export >> environment.yml
			#It may be necessary to remove some components such as VTK

			#Create a singularity def file (textfile) from below
			Bootstrap: docker

			From: continuumio/miniconda3

			%files
			environment.yml

			%post
			/opt/conda/bin/conda env update --file environment.yml --name base

			%runscript
			exec "$@"

			#Build container using singularity
			sudo singularity build conda_env.sif def_file.def

	===Nipype===		===Nipype===

Jstout: /* Nipype */

2020-08-21T15:24:23Z

Nipype

← Older revision		Revision as of 11:24, 21 August 2020
Line 8:		Line 8:
	===Nipype===		===Nipype===
	[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]		[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]

			[[File:Nipype pipeline.png\|center \| 400px]]

	===Git===		===Git===

Jstout: Initial

2020-08-21T15:05:56Z

Initial

New page

====Reproducible MEG Analysis====
MEG data collection and analysis requires a lot of interacting components (Stimulus delivery, logfiles, projector onset triggers, analog and digital triggering, electrophysiology measures, MRI coregistration...) that can lead to heterogeneous data and data quality. This makes pipeline analysis particularly difficult due to the variability in inputs. Additionally, with the multitude of techniques and metrics to evaluate MEG data (dipole/MNE/dSPM/beamformer/DICS beamformer/parametric stats/non-parametric stats ....) projects can have evolving analysis protocols. Knowing what preprocessing was performed on what datasets can often come down to an excel spreadsheet or text document. Since these projects can sometimes last multiple years, it is important to know what was done to produce a specific result and how to apply these analysis techniques to new incoming data.

===Containerized Computing Environment===
[https://sylabs.io/docs/ Singularity Containers]

===Nipype===
[https://nipype.readthedocs.io/en/latest/ Nipype Pipeline Toolbox]

===Git===

===DataLad===
[https://www.datalad.org/ "Datalad link"]</br>
[http://handbook.datalad.org/en/latest/ "Datalad Handbook"]