The DAN Lab

Software/Computer Protocols

Connecting to the DAN lab server (Research Drive – Amador-Noguez)

To connect to the DAN lab server, you must request access from David Stevenson, and you must be connected to the university’s network. If you’re attempting to connect a computer from a non-Wisc network, you can connect via a VPN. You can find out how to do this with the following link:

https://kb.wisc.edu/helpdesk/page.php?id=90370

Windows (Mac directions are below):

  1. Ensure you are connected to the university’s network before proceeding
  2. Open your file explorer

IMAGE 1

  1. Right click on your computer name to reveal the drop-down menu and click on “Map network drive…”

IMAGE 2

  1. Choose your desired letter for the Drive. Type (or copy & paste) the following address for the Folder:

\\research.drive.wisc.edu\amadornoguez

  1. If you are connecting to the lab server from a computer that will always be connected to the university’s network (such as a lab computer), keep the “Reconnect at sign-in” box checked. Otherwise, it’s recommended to uncheck that box for personal computers that are not regularly / automatically connected to the university’s network because your computer will reattempt to connect to the lab server every time you sign-in.
  2. Check the “Connect using different credential” box
  3. Click Finish

IMAGE 3

  1. Use your full Wisc account credentials to connect to the server (including the @wisc.edu) and click OK. You are now connected!

IMAGE 4

Mac:

  1. Ensure you are connected to the university’s network before proceeding.
  2. Click on “Go”

IMAGE 5

  1. Click “Connect to Server…”
  2. Type this into the drop down bar and push connect.
  3. Login with your wisc.edu full username and password.

IMAGE 6

  1. Now research drive should pop up in the Finder on the left side under “Locations”.

IMAGE 7

Last updated: 08/26/2021 Lauren Lucas

Running MZmine using scripts

Objective: This section describes the method for using scripts to process lipid data using MZmine instead of using the manual protocol.

  • Start MZmine. Open the “Project” menu and click on “Batch mode”:

IMAGE 8

  • Click “Load” and use the dialogue menu that pops up to open the batch script you want to run. For running negative mode samples only, select the batch script that has a name ending in “_negative”, for positive mode only select “_positive”, and to run both samples at once, select the script that says “_both”:

IMAGE 9

  • Select “replace” in the popup menu that appears:

IMAGE 10

  • You should get a screen that looks like the following:

IMAGE 11

  • Each of the entries in this list is one of the steps you manually perform in Julio’s walkthrough of MZmine. You can drag the lower-right corner of the box to make the box large enough to show the whole list of operations:

IMAGE 12

  • I used the script that runs both negative and positive files, so note that each operation exists in the list twice. The way I have it set up, first half of the operations work on the negative files and the second half on the positive files:

IMAGE 13

  • If you need to change parameters for any operations, you can highlight the operation in the list and click “Configure”. You will need to change a few things to make the batch process work at all. First, highlight the first instance of “Raw data import” and click “Configure”, then in the dialogue box that comes up, select and open the negative mode .RAW files you want to analyze. Next, select the first “Export to CSV file” entry and click “Configure”, then replace the text in the circled box with the path and filename you want the output of MZmine to be saved as (it should end in .csv and you probably want to pick a name that describes that it is the negative mode results) and click ok:

IMAGE 14

  • Now, because we are doing both files, we need to repeat the process in step 7, just for the positive mode files. So, highlight the second instance of “Raw data import” and click “Configure”, then in the dialogue box that comes up, select and open the positive mode .RAW files you want to analyze. Next, select the second “Export to CSV file” entry and click “Configure”, then replace the text in the circled box with the path and filename you want the output of MZmine to be saved as and click ok. Once you have done that, you should be returned to the screen here:

IMAGE 15

  • Click the “OK” button at the bottom of the screen to start the process. The results files will be saved where you specified (and will overwrite files with the same name). The popup window will close and you can monitor progress in the main window:

IMAGE 16

Quickstart guide for running INCA on the GLBRC servers

 

Make sure you have access to the GLBRC servers and network

If you would like to request access or need an account set up, email helpdesk@energy.wisc.edu to request access and CC Daniel on the email. If you run into any incorrectly denied connections or errors, report them to the same email address.

 

Connect to WEI VPN

To manage files or run scripts on the server, you must be connected to the WEI internet, either by being on location, or by connecting via the VPN.  To access the VPN, install GlobalProtect and use the settings below.

Install link: https://kb.wisc.edu/helpdesk/page.php?id=90370

Portal: weiglbrc.vpn.wisc.edu

Username: [GLBRC username]

Password: [GLBRC intranet password]

Once you are connected, it should look like this:

 IMAGE 17

 

Managing files on the GLBRC servers:

In order to manage files stored on the GLBRC server and upload/download files, you must be on the VPN and can use either SCP or SFTP protocols.  If unsure how to do this, download FileZilla (https://filezilla-project.org) if on Windows, Linux, or Mac OR WinSCP (http://winscp.net/eng/download.php) on Windows only. For more info, see the GLBRC IS’s  info page: https://intranet.wei.wisc.edu/information_technology/ITWiki/Home.aspx#navigating-cluster, but be aware that much of the information on that page is out of date, such as specific file paths or specific places to store data.

I used FileZilla for my connections, so if you are using that, use the settings below to connect to the file system on the server.

Host: scarcity-7.glbrc.org

(You may use any node of scarcity-X where X=[1 through 10], I am just using scarcity-7 as an example.)

Username: [GLBRC username]

Password: [GLBRC intranet password]

Port: 22

IMAGE 18

In the screenshot above, each section of FileZilla is marked with its function. You can navigate by typing a new path in “Current location on local computer” or “Current location on server” boxes or by clicking through the folder trees. Double-clicking on a file in the “Contents of selected local folder” or “Contents of selected server folder” folder will copy said file between the two folders; you may also drag and drop files. You can create or manage folders and files by right clicking in either “Current location on local computer” or “Current location on server” box.

Notes: The paths: /mnt/bigdata/linuxhome/[GLBRC username] and  /home/glbrc.org/[GLBRC username] refer to the same location in the filesystem. They are interchangeable and scripts or instructions might use either path.

You can change the automatic location accessed by FileZilla on your local machine in the settings, but it defaults to your user folder.

Connecting to WEI servers using SSH protocol

Use SSH protocol to connect to WEI compute servers. I used windows, so that method is first, but a brief Mac or Linux method is described below. You must be connected to the VPN for this.

On Windows, download PuTTY here: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html, then open PuTTY and connect using the settings:

Host Name: tbjacobson@scarcity-submit.glbrc.org

Port: 22

Connection type: SSH

IMAGE 19

You will be prompted for a password when the command line opens, use your GLBRC intranet password, then hit enter. While you are typing your password, no characters will show up in the command line, but the server still receives them. If everything worked, you should be greeted by the screen below:

IMAGE 20

On Mac or Linux, open terminal, then type:

ssh [GLBRC username]@scarcity-submit.glbrc.org

accept Host Key if prompted

enter [GLBRC password] when prompted

you should now see a screen similar to that shown for the windows process above

See the next section for a brief description of the commands you can use within PuTTY or terminal once you have connected via SSH protocol.

Commands you can use on the WEI servers

The WEI compute servers use the HTCondor framework, so if familiar with that system, you should be good to go and can directly enter commands or jobs as a series of arguments.  If not, see the manual here: https://research.cs.wisc.edu/htcondor/manual/, but some basic commands you might find useful for running INCA are listed below.  The general format is the command, followed by any options, followed by the object the command acts upon, each separated by a space.  In the list below, commands are in bold, options in italics, and the object is underlined.

cd [path] – allows you to change directories in the file system.  Typing cd without anything following it takes you up and out of the folder you are in.  Typing cd followed by a path, such as cd INCA/INCAv1.9/data_output will take you down into the data_output folder.  This can be important, as many scripts need to be run from a specific location. See screenshot example below, where I enter the INCA folder and then exit back to my home folder.

IMAGE 21

condor_q [GLBRC username] – Gives a summary of all jobs in the queue for the server and a list of jobs and the job IDs for things the user has submitted.

condor_q -analyze [job ID] – Gives detailed information about a specific job in the queue.

condor_rm [job ID] – Terminates a job.  Use this to kill jobs you don’t want to run anymore or any job you mistakenly submit or is stuck in the queue.

condor_submit [name of submit file.submit] – Used to submit jobs to the server in the form of a .submit file containing arguments – many of these have been put together by the IS team and can be retrieved from the https://gitpub.wei.wisc.edu/. Use the git command below to copy projects from gitpub over to your folders

git clone [address of project you want] – Use this to make a copy of stuff in gitpub in your own folders that you can use.

mkdir [name of desired folder] – creates a directory inside the current location with the name specified inside the brackets

Setting up INCA on the WEI servers (you will only need to do this the first time)

  • Connect to the WEI intranet using GlobalProtect
  • Make a folder (on the server) within /mnt/bigdata/linuxhome/[GLBRC username] that you want to work with INCA in (I called mine /mnt/bigdata/linuxhome/tbjacobson/INCA, so I will assume that for the rest of the instructions, but the name is arbitrary).
    • This is where the files you will work with will go and the output will appear
    • The easiest way to do this is probably FileZilla or the mkdir command
  • Connect via SSH using PuTTY or Terminal and cd into the folder you just created
    • For me, the command is cd INCA
  • Use the command git clone https://gitpub.wei.wisc.edu/jacek.kominek/inca-script to copy the INCA script that Jacek created into your folder
    • Read the readme that comes with the script or go to that webpage for more details
  • Using FileZilla, put the INCA install folder from your local computer inside the folder you just created on the server using FileZilla
    • As of 8/19/20, this folder is called INCAv1.9 and is usually located somewhere like C:\Program Files\Mathworks or C:\Program Files\Matlab
    • You can just drag and drop it from your computer into the “Contents of selected server folder” of FileZilla in the above screenshot
  • If you do not have a local installation of INCA, you can try using the folder on the server at \DAN Lab Business Folder\Protocols and Recipes\INCA scripts
  • Download the run_inca.submit file from your folder on the server and edit it to refer to the files in your folder, then reupload it to the server
    • Easiest way to download and edit is to doubleclick on it in FileZilla, then edit it using a text editor and drag and drop it back into the server on FileZilla
    • Make the following changes in the run_inca.submit file
      • Change the executable = line to read executable = /mnt/bigdata/linuxhome/[GLBRC username]/INCA/run_inca.sh
      • Change the –workdir_path and –out_path to refer to /mnt/bigdata/linuxhome/[GLBRC username]/INCA
      • Change —inca_path to refer to /mnt/bigdata/linuxhome/[GLBRC username]/INCA/INCAv1.9
        • This is the INCA install folder from your computer

Copies and example versions of the run_inca.submit, run_inca.sh, and run_inca.mat files are available, each set up to run (with my username and settings) on the server at \DAN Lab Business Folder\Protocols and Recipes\INCA scripts

 

Running INCA on the GLBRC servers

  • Use the INCA interface on your local computer to select reactions/measurements/datasets that you want to use in the solution, then save the model
    • Ideally, use a short name without any spaces or special characters
    • The file should be named something like mat
  • Upload your matlab file to the working INCA folder you created
  • Download and edit the run_inca.submit file from the server to refer to the file you just uploaded, then upload your edited version
    • Change the –model to refer to /mnt/bigdata/linuxhome/[GLBRC username]/INCA/mat
    • Make any other desired changes to cpu/memory requests in the submit file, then reupload
  • Through PuTTY or the terminal, use the command condor_submit run_inca.submit to send the instructions to the cluster
    • You should receive a confirmation and a job ID for the submitted job
  • Next use the command condor_q [GLBRC Username] to check the position of your jobs in the queue
    • Use condor_q -analyze [job ID] for details about a specific job
    • Use condor_rm [job ID] to terminate a job you want to stop or that gets stuck in the queue. If you don’t know why this happens, open a ticket with helpdesk@energy.wisc.edu
    • After you submit a job, you do not need to remain connected via PuTTY, FileZilla, or the VPN until you want to retrieve your data
  • Once complete, the job will output two files to the working INCA directory you made: mat and final_fluxes.xls that contain the results of the run
    • mat contains the model itself that can be opened in INCA on your local computer, but also contains the flux values and SSR values if you use the “open flux map” function in your local INCA window
    • xls contains the flux outputs in a simplified spreadsheet

Copies and example versions of the run_inca.submit, run_inca.sh, and run_inca.mat files are available, each set up to run (with my username and settings) on the server at \DAN Lab Business Folder\Protocols and Recipes\INCA scripts

Last updated: 12/02/2021 Tyler Jacobson