Difference between revisions of "Analysis AWS"

From MEG Core
Jump to navigation Jump to search
Line 1: Line 1:
''' For Workshop members without access to computing resources - Amazon Web Services servers can be used'''
 
   
 
''' For Workshop members without access to computing resources - Preconfigured Amazon Web Services servers can be used'''
To access the compute system:
 
  +
ssh -X -i ~/Downloads/AWS.pem ubuntu@IP_ADDRESS_PROVIDED
 
  +
== Accessing compute resources on AWS: ==
  +
Speak with Jeff Stout to get the current IP address and PEM key <br>
  +
${PATH} is the path to the PEM key provided to you
 
ssh -X -i ${PATH}/AWS.pem ubuntu@IP_ADDRESS_PROVIDED
  +
  +
== Jupyter Notebook Over a Remote Connection ==
  +
1) Run one terminal to start the notebook:
  +
ssh -Y $USER@$REMOTE_HOST
  +
cd $FOLDER_WITH_CODE
  +
#If necessary: conda activate $CONDA_ENV
  +
jupyter notebook --no-browser --port=8887
  +
  +
2) Run another terminal to forward the port to your local machine<br>
  +
<em>(Replace User and Remote_Host with the appropriate username and ip-address)</em>
  +
ssh -N -L localhost:8888:localhost:8887 $USER@$REMOTE_HOST
  +
  +
3) In Firefox/Chrome/Etc - Log into localhost:8888 <br>
  +
This will prompt you for a token<br>
  +
<br>
  +
4) In the password/token column, copy the token created in the first terminal<br>
  +
(It should look like the below - only copy the portion after the token= )
  +
http://localhost:8887/?token=####HERE_IS_THE_TOKEN_CODE---COPY_THIS_AND_PASTE_INTO_BROWSER
  +
  +
'''Biowulf Jupyter Notebook Connection'''<br>
  +
1) Run one terminal to access the persistent biowulf node and start jupyter notebook:
  +
#Connect to login node
  +
ssh -Y $USER@biowulf.nih.gov
  +
  +
spersist #This will startup a persistent node
  +
#Alternatively if a persistent node is already running, you can ssh into the persistent node
  +
  +
#In the persistent node
  +
conda activate $CONDA_ENVIRONMENT_NAME
  +
cd $FOLDER_WITH_CODE
  +
jupyter notebook --no-browser --port=8887
  +
  +
2) Run another terminal to forward the port to your local machine<br>
  +
<em>(Replace User and Remote_Host with the appropriate username and ip-address)</em>
  +
<em>(This is a two step process, the notebook port is sent to biowulf first, then to your local computer)</em>
  +
ssh -L localhost:8888:localhost:8888 $USER@biowulf.nih.gov
  +
#In the same ssh terminal from the line above this one, type the following
  +
ssh -N -L localhost:8888:localhost:8887 $USER@$PERSISTENT_NODE
  +
  +
3) In Firefox/Chrome/Etc - Log into localhost:8888 <br>
  +
This will prompt you for a token<br>
  +
<br>
  +
4) In the password/token column, copy the token created in the first terminal<br>
  +
(It should look like the below - only copy the portion after the token= )
  +
http://localhost:8887/?token=####HERE_IS_THE_TOKEN_CODE---COPY_THIS_AND_PASTE_INTO_BROWSER
  +
  +
  +
  +
   
 
''' Installed software ''' <br>
 
''' Installed software ''' <br>

Revision as of 09:43, 1 November 2019

For Workshop members without access to computing resources - Preconfigured Amazon Web Services servers can be used

Accessing compute resources on AWS:

Speak with Jeff Stout to get the current IP address and PEM key
${PATH} is the path to the PEM key provided to you

 ssh -X -i ${PATH}/AWS.pem ubuntu@IP_ADDRESS_PROVIDED

Jupyter Notebook Over a Remote Connection

1) Run one terminal to start the notebook:

 ssh -Y $USER@$REMOTE_HOST
 cd $FOLDER_WITH_CODE
 #If necessary: conda activate $CONDA_ENV
 jupyter notebook --no-browser --port=8887

2) Run another terminal to forward the port to your local machine
(Replace User and Remote_Host with the appropriate username and ip-address)

 ssh -N -L localhost:8888:localhost:8887 $USER@$REMOTE_HOST

3) In Firefox/Chrome/Etc - Log into localhost:8888
This will prompt you for a token

4) In the password/token column, copy the token created in the first terminal
(It should look like the below - only copy the portion after the token= )

 http://localhost:8887/?token=####HERE_IS_THE_TOKEN_CODE---COPY_THIS_AND_PASTE_INTO_BROWSER

Biowulf Jupyter Notebook Connection
1) Run one terminal to access the persistent biowulf node and start jupyter notebook:

 #Connect to login node
 ssh -Y $USER@biowulf.nih.gov
 
 spersist #This will startup a persistent node
    #Alternatively if a persistent node is already running, you can ssh into the persistent node
 #In the persistent node
 conda activate $CONDA_ENVIRONMENT_NAME
 cd $FOLDER_WITH_CODE
 jupyter notebook --no-browser --port=8887

2) Run another terminal to forward the port to your local machine
(Replace User and Remote_Host with the appropriate username and ip-address) (This is a two step process, the notebook port is sent to biowulf first, then to your local computer)

 ssh -L localhost:8888:localhost:8888 $USER@biowulf.nih.gov
 #In the same ssh terminal from the line above this one, type the following
 ssh -N -L localhost:8888:localhost:8887 $USER@$PERSISTENT_NODE

3) In Firefox/Chrome/Etc - Log into localhost:8888
This will prompt you for a token

4) In the password/token column, copy the token created in the first terminal
(It should look like the below - only copy the portion after the token= )

 http://localhost:8887/?token=####HERE_IS_THE_TOKEN_CODE---COPY_THIS_AND_PASTE_INTO_BROWSER



Installed software
Human Neocortical Neurosolver

 hnn
 #hnn is installed in the system level python >> may need to conda deactivate first

MNE python, Eelbrain, PyCTF

 conda activate workshop  (This environment has MNE, Eelbrain, and PyCtf installed)

CTF software

 singularity shell /opt/ctf/ctf.img
 $ctf_command
 singularity exec /opt/ctf/ctf.img $ctf_command

SAM version 5

 sam_cov ...
 sam_3d ...

Afni

 afni -dset $MRI_file



  1. Server Config
    1. google > fedora aws > select the AMI number for N.Virginia
    2. In AWS > launch instance > search for AMI number > (Under community) > Launch
      1. https://alt.fedoraproject.org/cloud/
  2. Determine an image (search for fedora in community)
  3. Create a keypair and download the key
    1. The download will be in the form of a .pem file
    2. We will need to distribute this key to the users
  4. Launch the instance
  5. Log into the instance using ssh


Conda installed Conda deactivate >> hnn Conda activate nih >> mne or eelbrain.....


  1. Freesurfer
  2. MNE python
  3. pyctf

Done # SAM v5

  1. Afni ###INSTALL ME neurodebian
  2. singularity ? >> CTF (need to use workstation version of fedora) << may need to compile for ubuntu or use 2.6
  3. Eelbrain (conda forge)
  4. Jupyter

Done# Forward sim HNN software

  1. FSL ###Install outside of neurodebian
  2. Brainstorm (compiled)


Matlab Based

  1. Brainstorm
  2. Fieldtrip