Teton: beginner's guide

Access to Teton

Teton is free to use for UW researchers. A faculty member (or you on your adviser's behalf) can request a new project on Teton from the ARCC folks for you to use (all users are part of one or more “projects”). There are forms to request a new project or to add a user to a project. All requests will need to be confirmed by the PI on the project. You can follow the steps below to connect to Teton once ARCC has set you up with an account.

Two-factor authentication

Logging into Teton requires two-factor authentication, which we need for many UWyo systems. Most users will likely have done this before; if you have not, you can click here to enroll in two-factor authentication at the University of Wyoming.

Teton is a UNIX system

The computers that make up Teton run a UNIX operating system (namely Red Hat Enterprise Linux Server, release 7.7 in April 2021).  There are many things that are consistent across different UNIX systems and an introduction to some of the ways to work with UNIX can be found in this introduction to UNIX that @Alex Buerkle wrote some time ago, but should still be relevant and useful.  It covers some of the concepts below, but is in printable format that might be useful to have at hand as a reference as you start working with UNIX.

ARCC has also created a Basic Linux for using HPC self-paced training on their WyoLearn space to get folks familiar with using Linux on Teton.

Connect to Teton

Terminal window access

Mac OS or other UNIX systems

Open the Terminal program (on MacOS, in /Applications/Utilities or from the Launcher), then type:

You will then be prompted to enter your UWyo password. Once you do so, hit Enter and it will look like nothing is happening, but you will receive a Duo push notification that you need to confirm in order to log in. Alternatively, you can enter your UWyo password, a comma, and a Duo token (a number) that you generate in the mobile app.

If you would like to use X11 forwarding (to use software that provides graphics), then when you login type:

To learn more about X11 forwarding, check out the section below on how to set up X11 forwarding. Alternatively, FastX is available and more responsive for remote use of a graphical desktop.

Open

Windows

If you are using Windows, you have several options to connect to Teton, ARCC has a couple of tutorials for doing so, or follow this guide on using PuTTY:

1. You can download PuTTY to access Teton. When you open PuTTY, you will get the PuTTY Configuration window (screenshot below). For the host name, type in username@teton.uwyo.edu. You can then hit enter, and you will get the terminal window with prompt to enter your password. You can also save your settings into a Saved Session. You can set up X11 configuration via PuTTY and have that automatically selected each time you log in (see X11 section below). Play around with the options and pick your favorite background and font colors, set up the host name, enter a name for this session right below "Saved Sessions", and click Save. Next time you can double-click that session name to load in with all of your preferred settings intact. 
   
   

   Open

   

2. If you have a Windows 10 machine, you can run Linux using Windows Subsystem for Linux. Once you have done that, you can log into Teton using the Ubuntu terminal as above for a UNIX machine. Click here for a tutorial on installing Windows Subsystem for Linux.

3. Windows users who use Google Chrome can install the Secure Shell Extension terminal emulator and SSH client . Once this extension is installed, you can type ssh username@teton.uwyo.edu into the address bar in Chrome. This will open the SSH extension, where you will be prompted to enter your password. Once you do so, hit Enter and it will look like nothing is happening, but you will receive a Duo push notification that you need to confirm in order to log in.

4. Another option is to install Git for Windows on your computer and use the Git bash terminal that comes with the download. This bash terminal allows you to run bash and use Unix commands on your Windows computer. The Git bash terminal can then be used to access Teton as above for a UNIX machine.

Graphical interface

FastX

Teton provides a fast interface to graphical window managers through FastX. This can be used most easily in a browser, but can also be run as a standalone FastX application for client computers.

A prerequisite is having a VPN connection to campus. This too can be obtained through a browser (connect to wyosecure.uwyo.edu and provide UW credentials), or by installing and running Pulse Secure locally (available through wyosecure).

X11 display forwarding

Alternatively, teton also supports X11 forwarding through ssh for graphics, though the responsiveness of applications will typically be low. Click here for more details on X11. Or, jump directly to install XQuartz if you have a Mac OS, or Xming or Cygwin if you have Windows. For a tutorial on setting up X11 forwarding with Windows Subsystem for Linux, check out X11 with Windows Subsystem for Linux. Typically users will prefer using FastX.

System organization and types of nodes

Teton consists of two login computers ("nodes"), many nodes for doing computation, and other specialized components.  The login nodes are the landing place for users who have logged in.  Typical usage involves logging to teton.uwyo.edu and requesting interactive or non-interactive jobs to be run using the job resource manager (SLURM).  Login nodes should not be used for computational heavy lifting (see the etiquette section below).

In 2019 the standard nodes ("Teton Regular") have 32 cores and 128 Gb RAM each. If you find that you need more RAM, e.g. for assembling genomes, you can use nodes with more installed RAM (presently BigMem nodes have 512 GB and HugeMem nodes have 1TB).  Likewise, if you need more cores within a single machine, or access to a GPU, there are corresponding nodes (details on the ARCC wiki).

SLURM – Our manager of compute resources

We use software to ensure users receive their fair share and priority for Teton usage.  This software is named SLURM and is commonly used on computer clusters (and is very similar in usage to other resource managers that you might come across elsewhere).  ARCC's wiki has a page with information on how to submit jobs that contains useful information.  Additionally, the Slurm command cheat sheet can be useful.

SLURM can be used to use Teton interactively or non-interactively. Login nodes should not be computational heavy lifting and instead SLURM should be used.

How to run an interactive job and get off the login node

When you first log on to Teton, you will be on a login node. This is ok if you are just navigating file directories, but you should get off the login node if you are doing anything more substantial, because your commands will be running on a system that typically has 10–20 other users at the same time. To start an interactive session, type:

where you replace project with the name of a project you have access to on Teton (e.g. compbio, passerinagenome) and set -t equal to the amount of time you need (format is day-hour:min).

This parts of this command are (salloc documentation):

• salloc Run an interactive slurm job.

• --account="project"  Associate resources used by this job to specified account. All users are associated with at least one project account.

• -t 0-01:00: Stay in interactive mode for an hour. Specifically, this is telling Teton/SLURM that you want to stay on a node for 0<days>-01<hours>:00<minutes>. Note that the maximum time for a job, without a special allocation, is 7 days (i.e., -t 7-00:00).

Here is another example:

This command specifies the number of nodes and threads that you are requesting, as well as 500GB memory is requested. If you use interactive nodes frequently, then you could add this command as an alias in your bash profile (here is a tutorial on how to do this) or you can possibly find in the history of previous commands (ctrl-r and begin typing part of the command you are searching for).

How to submit a non-interactive job with SLURM

Non-interactive jobs can be submitted two ways, either using the srun or sbatch command. The job will remain interactive (and the output will appear in your terminal window) if you use srun, while sbatch submits the job to a different compute node for processing. This means that sbatch allows you to log out of Teton while your script is running. It should also be used if you would like to continue to do other things without your terminal window being cluttered by the output of your script. In most cases, if you are submitting a script, you are likely to want to use sbatch.

With srun, you need to provide account information, time request, etc. on the command line when you submit the job. When using sbatch, these commands are included in the header to your shell script, as described below.

How to run a script using sbatch

Create a file and edit it using nano:

In the new text file, add commands to be run. The top line indicates that the script is to be interpreted as bash code. The following lines that begin with #SBATCH are instructions for SLURM, our scheduler and resource manager. These include at a minimum the project account name and time request. The other specifications are optional; for a full list of possibilities, type man sbatch in Teton (or consult the sbatch documentation). Below, we are specifying a name for the job (which makes it easier to check on your job as it is running), the number of nodes requested, and the number of threads requested (ntasks-per-node) for a multithreaded task and software that use multiple threads.

Save your script, then run your script with:

Submitting the script will request the number of nodes for the amount of time that you specified in your script. If the required computing resources are immediately available, then your job will start running immediately. If not, then the job will be queued until the resource becomes available. SLURM implements rules to share the teton system fairly and give priority to shorter jobs needing fewer resources, and from users who have fewer resources recently.

You can also check the status of all jobs that you have submitted by executing: squeue -u username

Each job will create an output file. To look at job progress, look at the file that is created in the folder where you submitted the job:

If you did not specify an output file and error file name, this file will be named using the SLURM job id. Otherwise, it will have the names that you specified. After the job is complete, you can query how long it took:

How to run a set of commands in a script using srun

1. Make a script (a text file with commands) called test.sh:

2. The test.sh script needs to be made executable:

3. Submit the job that is in the test.sh script to the SLURM scheduler:

Other handy commands

To view all jobs running on Teton:

The first command gives a summary of all of the jobs currently running, summarized by user and project. The second command summarizes your use of your disk quota. The third command provides a list of every job currently running, with information about the node it is running on, how long it has been running, etc.

To cancel one of your jobs

where you replace jobid with the job identification number given by squeue for the job that you would like to cancel. If you needed to cancel all of your currently running SLURM jobs, use scancel -u username

If you need to cancel multiple (but not all) running jobs, check out the tutorial on how to cancel a lot of jobs with Slurm.

How to navigate the file directories

Navigating the Teton filesystem is navigating a UNIX filesystem. We will go over a few basic commands that you will use regularly, but eventually you will want to consult UNIX help pages, books, or tutorials. If you like gamified learning, check out MIT's Terminus. Filenames, etc. are all case-sensitive.

If you have not learned this already, your use of UNIX will teach you that directories and filename should not contain spaces in their names. This is an instance of conformity being a good idea.

To list the contents of your current directory, type

To move to a new directory, use

where tmp is any existing name of a directory and /tmp is a full specification of the path to that directory (relative to the root of the filesystem /). For example, on Teton you would get to your project's space by

You can get to your gscratch space on Teton by

And you can access your home directory with

You can go up one directory by typing

or go up two directories by

This is super handy if you're way down in the depths of your folder structure and don't want to write out the entire path again. Or for setting relative paths in code.

If you are in your project directory and on the login node, you will see that the prompt has changed to something like this:

This means that you are in your project folder. Type ls to see what's in there and you can move directly down into a directory. Let's imagine that you have a folder in your project directory called scripts. To get there, simply type

You don't have to put the entire path because you are already in the directory that contains scripts. You will be using the commands cd and ls constantly. Other handy commands are mv to move a file (renaming is a form of moving, so use mv as the standard way to rename; the command rename is more full-featured but is not on all UNIX systems), cp to copy a file, touch to create an empty file (many editors will do this for you), mkdir to create a directory, less to look in a potentially enormous file without ‘opening’ the file and bringing it all into memory.

Other friends for working with text files are: head, tail, cat, cut, wc, sort, uniq, their use in combination by piping output of one to the next, and a world of powerful others.

What are gscratch, project, and home?

When you get a Teton account, you are given some space in /home/username and much more space in /project/yourproject and /gscratch/username. You can get to this space by typing cd, cd ~/ or cd /home/username. Your home directory is a small space, but useful for storing scripts or software. If you don't set a path when moving or creating files, the default is often your home space. Check here first if you lose something.

Project accounts have space in /project/projectname, where projectname is the name of your project–e.g. WagnerLab, passerinagenome. Some groups have named the project after their lab, whereas others have named the project after the actual research project.

There are two options for scratch space global scratch called gscratch and local scratch called lscratch. Scratch space is temporary storage that is subject to ARCC’s purge policy.

• global scratch is a large storage allocation intended for working data that is always on the fastest storage available. It is also accessible from every node on the system.

• Local scratch is storage that is dedicated to the node you are working on. This storage only lasts for the duration of your job, but provides better read/write speeds if that is required in your workflow.

Additional details can be found on the ARCC wiki on Teton’s filesystem.

How to move files to and from Teton

Note: Before you get started, make sure you don't have any SPACES or PERIODS (except for the period preceding your prefix) in your file names! This could cause you problems you don't necessarily want. 

Command Line Interface (Mac, Linux)

The format these commands follow is the source followed by destination.

For example, if you want to move just one file called 'example.txt' from your Desktop to a project directory on Teton: 

Similarly if you want to move a directory called 'inputdata' from your Documents to your home directory on Teton instead:

If you want to put it somewhere else on Teton, modify the path after the colon.

To move something from Teton to your local machine, first open a local terminal window. If you had a text file called example.txt in your home directory and you wanted to move it to your Desktop, you would then type:

You will then be required to login and authenticate with the usual two-factor authentication.

You will want to use scp when you have small files to transfer to and from Teton but for larger directories, it is recommended to use rsync since this command checks the timestamps and only transfers the files that have been changed. This will speed up transfers of large files. Here is a resource from ARCC for more stuff: ARCC's page on using UNIX commands to transfer data

There may be instances where you'd want to download files from the web directly onto your project directory in Teton. Here are a couple of command line tools you can use from where you want your file to be downloaded:

The -O option allows you to give your downloaded file a different name. If you want to keep the same name as the remote file, you can omit this option. 

The option to use Firefox as a browser from Teton is also available. This helps you navigate to a web page from which you can download the file with clicks and links! For this all you need to do is type in firefox at the prompt and this will open up the program. From here, you can copy and paste your download link in the search bar and you're good to go.

You can also use the curl program to do the same thing as wget:

Globus is a browser-based option that is easy but not the most efficient–and I (Libby) use it all the time. ARCC prefers that you use Globus to move large files to/from Teton or petaLibrary. The setup takes a while but once you've designated your laptop/work computer as an 'endpoint', you can move large files to and from Teton with less effort. 

Windows users

If you are on a Windows machine, you can either install a bash shell or download a software specifically for transferring files. A bash shell, as described above, will give you all the same functionality as a UNIX machine and you can use the same commands from before. Some options for GUI-based file transfer software are WinSCP or CyberDuck. These GUIs offer the option to drag-and-drop the files to and from a remote location, such as Teton or the petaLibrary. This is suitable for small files but not for larger files since it does slow down the transfer. 

If you have PuTTY installed, you can use the pscp command in Windows command prompt to copy small files. You can pull up command prompt by typing “cmd” in the search bar at Start. Using the example above, if you want to move just one file called 'example.txt' from your Desktop to a project directory on Teton, you can type:

You will be prompted to provide your login credentials. When copying from a Windows machine, be careful to use quotation marks around any file locations that contain spaces.

Modules on Teton

The Teton HPC system uses modules as a way of organizing software that many users may use. A module is software that has been installed as a system package so that all users of Teton can access it. There are some modules that are automatically loaded when you login to Teton, and others that require that you load them for use. ARCC has a wiki page explaining the module managing system (Lmod), which can be found here. Software is organized hierarchically by compiler. The supported compilers are Intel, GCC, and PGI.

Modules are installed by ARCC staff. If there is a piece of software missing on Teton that you need to use--and you think that this software will be useful for several other people to have on Teton as well--you can submit a software request to request that this software be installed. If you aren't sure that others will use the software, you can also install it locally for yourself by following this ARCC guide.

The modules available in your Teton session will vary based on the compiler that you have loaded. To view currently loaded modules, type

To view all available modules based on the modules that you already have loaded, type

Searching for modules is a great way to find out if software has been installed as a system package.  To search for a specific module, type

where "software" is the software name you're looking for. This command will output the name of the module, the versions available for loading, and any dependencies that must be loaded prior to loading the given module. You can also type a partial name to search for all software containing a specific phrase, or use any regular expressions to search for specific components of a module name. Alternatively, you can use 'module spider' without anything after it to return a list of all currently available modules. If you still are unsure about which module is the one you want, you can search through all module names and brief descriptions using

where 'term' is the keyword you want to search for. Once you have identified the name of a module that you want to load, you can load this module using

If you get the error "Lmod has detected the following error: These module(s) exist but cannot be loaded as requested", use 'module spider modulename' to find the modules you must load before loading the desired module.

You can also unload modules loaded in error or no longer needed in a session using

If you find yourself frequently loading a specific suite of modules, you can save these modules as a collection that can be loaded by the package name. To do this, load your desired modules and then type

where 'name' is what you would like the collection of modules saved as. You can then load this collection of modules using

If you can't remember what you saved your module, 'module savelist' will print out a list of all of your saved collections.

Help pages for specific modules:

Exiting Teton

Once you're done using Teton, you'll likely want to exit from the system. At command prompt, simply type exit and hit return.  This is preferable to simply closing the Terminal window using a button, as it give the shell an opportunity to exit properly.

Asking additional questions and getting help

If you still have questions related to using Teton, or other data science questions, please use the Spring semester 2019 - Open question and answer thread. Alternatively, you can add a task or ask a question on the Development of Teton's Beginner's Guide board. 

A note on Teton usage etiquette

Since Teton is widely used by many research groups, all users should be aware of best practices for polite usage. Rules of etiquette are inspired by the need for equitable distribution of resources among users.

Specific usage etiquette is subject to change, but here are a few guidelines that will likely remain constant and should be kept in mind.

First, do not use the login nodes for any serious computational work. Doing so slows down the nodes for other users. If interactive computing is desired then it is possible to move off the login node to a different node for such a session (see the section above titled: “'How to run an interactive job and get off the login node”

Second, be realistic in the requirements for a computing task. A job that only requires a few Gb of RAM should not be run on the hugeMem nodes, since that renders those nodes inaccessible to other users that may legitimately need much greater memory capacity for their project. Similarly, be wary of asking for exceptionally long wall times for jobs that likely will run quickly. SLURM captures information on individual user requests and attempts to divvy up resources in a fair way. Thus, if one continually asks for extreme amounts of resources, one will eventually find it harder to have jobs be accepted. We are lucky in that most of the time there are plenty of resources available on Teton, but as more researchers use the system it will become more important to make sensible resource requests.

Third, if working as part of a collaborative project, hard disk space allotted to the project can become limited. It is thus best practice to remove intermediate files to avoid wasting time and disk space backing those files up. Instead, place intermediate files in your scratch disk and delete them when they are sure to be of no further use.

Related articles