MEG analysis on Biowulf
Biowulf brief intro
Biowulf (biowulf.nih.gov) is the head node of the Biowulf cluster at NIH - https://hpc.nih.gov/docs/userguide.html
#For processing of data - reserve an sinteractive session sinteractive --mem=6G --cpus-per-task=4 #Depending on your usage, you may need more --mem and more cpu cores. Also for jupyter notebook, you will need the --tunnel option
Helix - is the storage server attached to the biowulf cluster. !!Do not upload/download data directly to biowulf - use helix!!
#Use scp for transfer or rsync -av (slower but some benefits) scp -r ./my_local_results ${USERNAME}@helix.nih.gov:/data/mydatafolder/....
#Getting results from biowulf cluster scp -r ${USERNAME}@helix.nih.gov:/data/mydatafolder/... ${PATH_OFSTUFF}/my_local_stuff
Analysis of data should not be performed on the biowulf head node, but run through an sinteractive node or swarm process.
To start with, there are a limited number of commands loaded on the system. To access more programs use module load. To search, use module spider.
e.g. module load afni
Configuring your bash shell environment
If editing your bashrc -- open two terminals in biowulf. If you misconfigure your .bashrc, you will not be able to log into biowulf. Having two terminals open allows you to fix anything that errors out.
Edit .bashrc file in your home drive
umask 002 #Gives automatic group permissions to every file you create -- very very helpful for working with your team #Add modules bin path to access the MEG modules PATH=/data/MEGmodules/bin:$PATH ## Set up some aliases, so you don't have to type these out alias sinteractive_small='sinteractive --mem=8G --cpus-per-task=4 --gres=lscratch:30' alias sinteractive_medium='sinteractive --mem=16G --cpus-per-task=12 --gres=lscratch:100' alias sinteractive_large='sinteractive --mem=24G --cpus-per-task=32 --gres=lscratch:150'
Edit .bash_profile in your home drive
#Set your default group (normally your default is your userID - which isn't helpful for your group #Type `groups` to see which groups you are part of newgrp <<YOUR GROUP ID>>
SAM MEG Data Analysis
module load afni module load ctf module load samsrcv3/20180713-c5e1042
MNE python data analysis
To Access Additional MEG modules
#Add the following line to your ${HOME}/.bashrc module use --append /data/MEGmodules/modulefiles
Load MNE modules
#The module will not load on the biowulf head-node because freesurfer loads #Create an sinteractive or spersist node - adjust memory and cpus core number accordingly sinteractive --mem=6G --cpus-per-task=4 module load mne/0.24.1 # OR module load mne <<-- defaults to most current version ipython import mne
Load Commandline MNE scripts for Processing Data
!Currently in Beta Testing!
[stoutjd@cn1023 modules]$ module list No modules loaded [stoutjd@cn1023 modules]$ module load mne_scripts [+] Loading freesurfer 7.1.1 on cn1023 [+] Loading mne 0.24.1 ... [+] Loading mne_scripts 0.1_dev ... Available: spatiotemporal_clustering_stats.py [stoutjd@cn1023 modules]$ spatiotemporal_clustering_stats.py -h usage: spatiotemporal_clustering_stats.py [-h] [-topdir TOPDIR] [-search SEARCH] [-outfname OUTFNAME] [-subjects_dir SUBJECTS_DIR] options: -h, --help show this help message and exit -topdir TOPDIR The directory w/ stc files -search SEARCH The search term to find the specific stc files for clustering -outfname OUTFNAME Output filename for the cluster nifti file -subjects_dir SUBJECTS_DIR Freesurfer subjects dir. Will download freesurfer average if not already present. Defaults to os.environ['SUBJECTS_DIR'] if not provided
MNE bids creation and MNE bids pipeline processing on biowulf
Start interactive session with scratch to render visualization offscreen
sinteractive --mem=6G --cpus-per-task=4 --gres=lscratch:50
Create BIDS data from data off the scanner
module load mne_scripts make_meg_bids.py -meg_input_dir MEGFOLDER -mri_brik AFNI_COREGED+orig.BRIK
Process data using MNE Bids Pipeline
module purge module load mne_bids_pipeline mne-bids-pipeline-run.py --config=CONFIG.py ##Optional --steps=preprocessing,source --subject=SUBJECTID(without sub-)
Best Practices for Group Data Preprocessing
Process your project data
Make your python script commandline callable
#It is typical to run 1 subject per commandline call and to parrallelize over subjects in the swarm file -Use argpase to manually build a full commandline call with keyword inputs and function description -Use fire or click to automatically create a commandline call based on function inputs -Use sys.argv[] to create a simple commandline input (sys.argv[0] is the filename - argv[1] is the first argument - argv[2] is second...)
Helpful Hints: -Use the meg dataset as an input, inside the python module - use the meg dataset to extract the subject ID : filename = os.path.basename(meg_dataset) subjid = filename.split('_')[0] OR if you have a custom ID subjid = filename[0:#characters] -Build output filenames using f-strings: outfile_base = f'{subjid}_{taskname}.nii' outfilename = os.path.join(outdir, outfile_base)
Build, test, and submit your swarm file
for i in ${GROUP_FOLDER}/*.ds; do echo my_process.py -in1 input1 -in2 -input2 -dataset $i >> swarm_file_preprocess.sh ; done #Make sure the process runs on at least one subject/dataset #This will run the last line of the swarmfile $(tail -1 ./swarm_file_preprocess.sh) #Verify the results on the single subject / possibly look at how much RAM / CPU was used before submitting the full batch to swarm swarm -f ./swarm_file_preprocess.sh -g ${GigsOfRAM} -t ${CPUcores} # -b ${How many subjects to run in row on 1 computer} - can be useful if you have a fast process
ADVANCED: Making your own python module
Build the python conda environment
It is recommended to create an install script so that this can be sent to a slurm job
# Load conda - if set up according to the HPC page, this should work source /data/${USER}/conda/etc/profile.d/conda.sh; conda activate base # echo mamba create -p ${PATH_TO_OUTPUT} condaPackage1 condaPackage2 conda-forge::condaForgePackage1 -y > installFile.sh # Make sure to include the -y or the job will hang waiting for user response # Also make sure you have an active conda prompt when submitting the swarm, or else it will fail echo mamba create -p /data/ML_MEG/python_modules/mne0.24.1 jupyter ipython conda-forge::mne -y > python_install.sh swarm -f ./python_install.sh -g 4 -t 4
Make a module file
To display most of the contents of a module file run
module display python #For the python module
Output:
---------------------------------------------------------------------------------- /usr/local/lmod/modulefiles/python/3.8.lua: ---------------------------------------------------------------------------------- family("python") prepend_path("PATH","/usr/local/Anaconda/envs/py3.8/bin") pushenv("OMP_NUM_THREADS","1")
Copy Template to your module folder
#MyModule is the family name of the code / ${Version}.lua cp /usr/local/lmod/modulefiles/python/3.8.lua ${myModuleFilesDir}/${MyModule}/0.1.lua
Add module files to the search path
module use --append ${PathToUserModuleFiles}
Final Step Load Your Module
# !! The module load process does not give good feedback that it doesn't load - make sure the path in lua file is correct !! module load ${MyModuleName}
#Example [$USERd@$NODE python_modules]$ module load mne [$USERd@$NODE python_modules]$ ipython Python 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21:24:11) Type 'copyright', 'credits' or 'license' for more information IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: import mne In [2]: mne.__path__ Out[2]: ['/data/ML_MEG/python_modules/mne0.24.1/lib/python3.9/site-packages/mne']