This guide provides the basics of connecting to and using the Talapas HPC cluster.

Getting a Talapas Account

If you don't already have an account, see the Request Access page to get one.

Logging on to Talapas

For UO users, your username on Talapas will be your Duck ID.  (That is, if your email address is alice@uoregon.edu, your Talapas username will be "alice".)  Your password is the same university-wide, and can be managed at the UO the UO password reset page.

For non-UO users, you will have received a username and password in the email granting you access.

Talapas currently has two four login nodes:

• login1.talapas.uoregon.edu
• login2.talapas-ln1.uoregon.edutalapas-ln2
• login3.talapas.uoregon.edu
• login4.talapas.uoregon.edu

These hosts are entirely equivalent.

  You can use whichever seems less busy, or use hostname talapas-login  but the recommended method is by logging in via the load balancer:

         login.talapas.uoregon.edu

to be sent to one randomlylogged into one of the four nodes with the least amount of traffic.

If you are logging in from a Linux or Mac OS X workstation, open a terminal and type

ssh myusername@talapas-ln1myusername@login.talapas.uoregon.edu

If you are logging in from Windows, download an SSH client like Putty or MobaXterm and do the equivalent.

...

If you're accessing Talapas from Linux or Mac OS X, you can use scp or rsync to  to transfer files.  For example, type

scp chr1.fasta myusername@talapas-ln1myusername@login[1-4].uoregon.edu:.

to copy the named file to your Talapas home directory in either of the 4 login nodes (login1,...,login4)

There are also GUI tools available for file transfer via SCP or SFTP.  Filezilla is available for all common platforms.

Also,  /wiki/spaces/HPC/pages/1769536 Globus is available, and is particularly useful for the transfer of large files.

...

Here's a link to our wiki page, How-to: Use Globus

Storing your Data

As a Talapas user, you have these directories available for storing and manipulating your files.

...

See Storage for more details.

Software

Read about how to find software on our Software page.  Read on for the absolute basics.

Talapas uses the Lmod environment module software to provide access to the various installed software packages.  Type

...

See How-to Use LMOD for more details.

Running a Batch Job

Talapas uses the SLURM job scheduler and resource manager to provide a way to submit large computational tasks to the cluster in a batch fashion.

As a basic example, here is a simple batch script to run the hostname command on one compute node.  To run it, first create a file hostname.batch containing these lines:

#!/bin/bash
#SBATCH --partition=short       ### queue to submit to
#SBATCH --job-name=myhostjob    ### job name
#SBATCH --output=hostname.out   ### file in which to store job stdout
#SBATCH --error=hostname.err    ### file in which to store job stderr
#SBATCH --time=5                ### wall-clock time limit, in minutes
#SBATCH --mem=100M              ### memory limit, per cpu, in MB
#SBATCH --nodes=1               ### number of nodes to use
#SBATCH --ntasks-per-node=1     ### number of tasks to launch per node
#SBATCH --cpus-per-task=1       ### number of cores for each task

hostname

Then, to actually submit the job for execution, type

sbatch hostname.batch

When a job is submitted, SLURM considers all of the resources requested (e.g., CPU cores, memory, time, GPUs) together with the set of currently running and queued jobs.  Once compute nodes are available to run your job, it will be run.

Note that your job will not run until the resources you have requested are available.  This means that you may be able to get your job to run sooner by requesting fewer resources.  For example, if your job will run in 1000MB, it's better not to request 10000MB, as it may take significantly longer for those resources to become available.

Batch jobs are run in a manner that's not dependent on your current terminal, so once a job is submitted, it no longer depends on your being logged in, etc.  Job output will be available in the specified files when it has completed.  You can monitor your job's progress using the squeue and sacct commands.  For example

...

...

SLURM Accounts

In order to ensure correct service charges, all SLURM jobs on Talapas require that an account be specified.  You can do this explicitly using the --account (or -A) option to sbatch or srun.  This is shown in the first example below, but omitted from the rest for clarity.  Alternatively, you can set the environment variables SLURM_ACCOUNT and SBATCH_ACCOUNT in your ~/.bash_profile if you like.

If your SLURM job invokes subsidiary calls to 'sbatch', you might also need to include lines like this in your batch script to make sure that the subsidiary 'sbatch' calls also see the account information:

export SLURM_ACCOUNT=$SLURM_JOB_ACCOUNT
export SBATCH_ACCOUNT=$SLURM_JOB_ACCOUNT

Note that if you a member of multiple PIRGs, it is your responsibility to specify the correct account for each job.

If you're not sure what your PIRG/account name is, you can use the groups command.  It will show a list of groups that you're in.  A few are system groups that you cannot charge to (e.g., talapas, gaussian).  But within that list you should recognize the PIRGs that you are a member of.

Running a Batch Job

Talapas uses the SLURM job scheduler and resource manager to provide a way to submit large computational tasks to the cluster in a batch fashion.

As a basic example, here is a simple batch script to run the hostname command on one compute node.  To run it, first create a file hostname.batch containing these lines:

#!/bin/bash
#SBATCH --account=<myaccount>   ### change this to your actual account for charging
#SBATCH --partition=short       ### queue to submit to
#SBATCH --job-name=myhostjob    ### job name
#SBATCH --output=hostname.out   ### file in which to store job stdout
#SBATCH --error=hostname.err    ### file in which to store job stderr
#SBATCH --time=5                ### wall-clock time limit, in minutes
#SBATCH --mem=100M              ### memory limit per node, in MB
#SBATCH --nodes=1               ### number of nodes to use
#SBATCH --ntasks-per-node=1     ### number of tasks to launch per node
#SBATCH --cpus-per-task=1       ### number of cores for each task

hostname

Then, to actually submit the job for execution, type

sbatch hostname.batch

When a job is submitted, SLURM considers all of the resources requested (e.g., CPU cores, memory, time, GPUs) together with the set of currently running and queued jobs.  Once compute nodes are available to run your job, it will be run.

Note that your job will not run until the resources you have requested are available.  This means that you may be able to get your job to run sooner by requesting fewer resources.  For example, if your job will run in 1000MB, it's better not to request 10000MB, as it may take significantly longer for those resources to become available.

Batch jobs are run in a manner that's not dependent on your current terminal, so once a job is submitted, it no longer depends on your being logged in, etc.  Job output will be available in the specified files when it has completed.  You can monitor your job's progress using the squeue and sacct commands.  For example

$ sbatch hostname.batch
Submitted batch job 12345

# checking with squeue, we see that the job has started running (state is 'R')
$ squeue -j 12345
   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    12345   short     myhostjob   joec  R       5:08      1 n003

# later, we see that the job has successfully completed
$ sacct -j 12345
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
12345       myhostjob+      short    joegrp         1     COMPLETED      0:0

If needed, you can also kill a job using its job id.  This will terminate it if it has already started running, or remove it from the queue if it hasn't started yet.

scancel 12345

...

12345.batch      batch               joegrp         1     COMPLETED      0:0
12345.exte+     extern               joegrp         1     COMPLETED      0:0

If needed, you can also kill a job using its job id.  This will terminate it if it has already started running, or remove it from the queue if it hasn't started yet.

scancel 12345

SLURM uses a number of queues, which it calls partitions, to run jobs with different properties.  Normal jobs will use the short partition, but there are other partitions for jobs that need to run longer than a day, need more memory, or need to use GPUs, for example.  You can use the sinfo command to see detailed information about the partitions and their states.

...

srun --account=hpcrcf --pty --partition=short --mem=1024M --time=240 bash

...

The above command is good for running single programs.  If you'll be doing something that will invoke multiple parallel processes or threads, like a parallel make or multi-threaded program, you might want to add a --cpus-per-task=N flag to allocate more cores.

...

  Note that on Talapas, using --ntasks (or -n) instead of --cpus-per-task (or -c) is virtually always the wrong thing to do, and will make your session run more slowly.

Running Graphical Interactive Jobs

If your interactive job will involve a program like rstudio, which uses X11 to provide graphical output, you can use a command like this there are several options.  You can use the srun --x11 flag like so to forward your X11 connection back to your workstation X server.

xrunsrun --account=hpcrcf --x11 --pty --partition=short --mem=1024M4G --time=4:00 bash

(Although the other parameters are similar, note that time is in hours rather than minutes for this command.)

In order for this to work correctly, you will need to be running an X server on your local workstation, and you'll have to forward X traffic when you connect to a login node.  This is typically done by using ssh -Y or its equivalent.

Requesting Space and Time

For every SLURM job you should specify the amount of memory the job needs (with --mem and related flags) and the amount of time it needs (with --time).  If you don't do so, default values are used, but these are often less than ideal.

As a general rule, the fewer resources your job requires, the sooner it will run.  This is because smaller/shorter jobs are easier for SLURM to schedule sooner.  (Requesting fewer resources also benefits other users by allowing SLURM to schedule cluster resources more efficiently.)  That said, if you specify less space or time than your job needs, it will be killed before it can complete, since these specifications are enforced.  So, you want to err somewhat on the high side.  For any given application, you might have to experiment some to get this right.

For memory, the default is to be allocated a proportional amount of available RAM from the node.  So, for example, if you request one CPU core on a node that has 28 cores and 104GB of RAM, the default will be about 3.7GB of memory.  (A corollary is that if you request all of the cores for a node, you'll get all of the available RAM by default.)  Some nodes have more cores and/or RAM, so this may vary, but 3.7GB is currently the smallest default for all Talapas nodes.  See Machine Specifications for more detailed information about the available nodes.

If your job needs more than the default, you must explicitly specify a larger value.  Alternatively, if your job needs less, you might want to specify less, which will increase the odds that it will be scheduled sooner.

For time, the default varies by partition, but is generally the maximum available for the partition.  For the short partition, the default is currently 24 hours.  For the long partition, it's currently two weeks.  If your job will take significantly less time, you can specify a shorter duration, to increase the odds that it will be scheduled sooner.

In order to help estimate your job's requirements, you can use sacct to see what prior jobs required

sacct -j JOBID --format=JobID,JobName,ReqMem,MaxRSS,Elapsed

where JOBID is the numeric ID of a job that previously complete.  This will produce output like this

sacct -j 301111 --format=JobID,JobName,ReqMem,MaxRSS,Elapsed
       JobID    JobName     ReqMem     MaxRSS    Elapsed
------------ ---------- ---------- ---------- ----------
301111        myjobname     3800Mc              16:00:28
301111.batch      batch     3800Mc    197180K   16:00:30
301111.exte+     extern     3800Mc      2172K   16:00:29

This job used about 193MB (197180K) of RAM at its peak, and ran for a bit over 16 hours.

See Memory for more details.For some cases, the alternative script xrun works better.

xrun --account=hpcrcf --partition=short --mem=1024 --time=4 bash

Note that this script supports only a subset of the srun flags and has some quirks.  The --mem value must be a whole number and is in megabytes.  The --time value is in HH:MM format, but if only a whole number is specified, it is take as hours (rather than minutes).

In order for this to work correctly, you will need to be running an X server on your local workstation, and you'll have to forward X traffic when you connect to a login node.  This is typically done by using ssh -Y or its equivalent.

The final option, which is far simpler if it works for your case, is to simply start a desktop session using Open OnDemand.  This does not require an X11 server on your workstation, but rather runs the server on the compute node, forwarding traffic via a web browser-based VNC client.  See the Open OnDemand page for more details.

Requesting Space and Time

For every SLURM job you should specify the amount of memory the job needs (with --mem and related flags) and the amount of time it needs (with --time).  If you don't do so, default values are used, but these are often less than ideal.

As a general rule, the fewer resources your job requires, the sooner it will run.  This is because smaller/shorter jobs are easier for SLURM to schedule sooner.  (Requesting fewer resources also benefits other users by allowing SLURM to schedule cluster resources more efficiently.)  That said, if you specify less space or time than your job needs, it will be killed before it can complete, since these specifications are enforced.  So, you want to err somewhat on the high side.  For any given application, you might have to experiment some to get this right.

For memory, the default is to be allocated a proportional amount of available RAM from the node.  So, for example, if you request one CPU core on a node that has 28 cores and 104GB of RAM, the default will be about 3.7GB of memory.  (A corollary is that if you request all of the cores for a node, you'll get all of the available RAM by default.)  Some nodes have more cores and/or RAM, so this may vary, but 3.7GB is currently the smallest default for all Talapas nodes.  See Machine Specifications for more detailed information about the available nodes.

If your job needs more than the default, you must explicitly specify a larger value.  Alternatively, if your job needs less, you might want to specify less, which will increase the odds that it will be scheduled sooner.

For time, the default varies by partition, but is generally the maximum available for the partition.  For the short partition, the default is currently 24 hours.  For the long partition, it's currently two weeks.  If your job will take significantly less time, you can specify a shorter duration, to increase the odds that it will be scheduled sooner.

In order to help estimate your job's requirements, you can use sacct to see what prior jobs required

sacct -j JOBID --format=JobID,JobName,ReqMem,MaxRSS,Elapsed

where JOBID is the numeric ID of a job that previously complete.  This will produce output like this

sacct -j 301111 --format=JobID,JobName,ReqMem,MaxRSS,Elapsed
       JobID    JobName     ReqMem     MaxRSS    Elapsed
------------ ---------- ---------- ---------- ----------
301111        myjobname     3800Mc              16:00:28
301111.batch      batch     3800Mc    197180K   16:00:30
301111.exte+     extern     3800Mc      2172K   16:00:29

This job used about 193MB (197180K) of RAM at its peak and ran for a bit over 16 hours.

There's also a helper script 'seff' that provides similar information and is a bit easier to type.  Just say "seff JOBID".

See Memory for more details.

Special Handling for Quarterly Maintenance Events

Approximately once per quarter, Talapas is taken down for maintenance, to update software and hardware, make repairs, etc.  To minimize the disruption, SLURM reservations are added to "block out" those times.  If you take no special steps, a job request that you make that runs into a maintenance window will be held and run after the window.

This is often what you'd want.  But, sometimes you might want a job to start right away, even if the entire optimal running time is not available.  You can specify this using the "--min-time" flag to specify a minimum acceptable time limit for the job.

...
#SBATCH --time=0-8:00:00
#SBATCH --time-min=0-1:00:00
...

Using the above flag, you'll normally get a time limit of eight hours.  But, if there are reservations that would delay the start of your job, SLURM will consider any  time limit down to one hour to be acceptable and start the job immediately.  (SLURM will prefer the largest available time limit.)

Being a Good Cluster Citizen

...