Skip to content

TinkerCliffs

ARC's official documentation

https://www.docs.arc.vt.edu/

Hint: Words in [] brackets (example: [PID]) mean that you have to substitute something and you should omit the square brackets.

Warning: Never run long calculations on a login node! Run a job on a compute node!

  • a login node is where you conntect with the ssh [PID]@tinkercliffs1@arc.vt.edu command
  • a compute node is accessed from a login node by either submitting a job script or by starting an interactive job

Terminal

First: Open the terminal on your computer (do not use the ARC OnDemand website, because it has bugs).

SSH

Connect to TinkerCliffs using ssh (replace [PID]): 

ssh [PID]@tinkercliffs1@arc.vt.edu
to connect to the first login node or the second login node with: 
ssh [PID]@tinkercliffs2@arc.vt.edu

Run a compiled program on compute node

TinkerCliffs is a shared resource with many users.

Never run your program on the login node! (don't do this: ./[program name])

Instead use the slurm scheduler as shown next. See also ARC Documentation on slurm

Submit job script to queue

Submit script to the slurm scheduler: 

sbatch submission.sh

An mock-up job script is 

#!/bin/bash
#SBATCH -J hello-world          # Name of the job
#SBATCH --account=cmda3634_rjh  # Account allocation
#SBATCH --partition=dev_q       # Partition of the cluster (dev_q, normal_q)
#SBATCH --nodes=1               # Number of compute nodes
#SBATCH --ntasks-per-node=1     # Number of processes
#SBATCH --cpus-per-task=1       # Number of CPU cores per process
#SBATCH --time=00:10:00         # Runtime limit of 10 minutes

echo "Hello world from ..."
hostname

Notes on jobs

  • The jobs submitted via the sbatch command might take some time waiting in the queue, because the TinkerCliffs machine is currently busy with jobs from other users. While your job is waiting in the queue, there won't be any output slurm files.
  • You can submit a job and you don't need to keep your terminal open.
  • There might be a delay between the slurm output file being created and the first output from your program being printed.

Useful commands for checking on jobs

  • tail -f slurm-XYZ.out will continuously show new lines as they get output into the file.
  • squeue -u [PID] will show you the status of your submitted jobs. In the table shown, "PD" means pending job, "R" means running job, and finished jobs do not appear.
  • watch -n 5 squeue -u [PID] will monitor the command squeue -u [PID] and refresh every 5 seconds.
  • scancel [JOBID] (with the number of the job shown with squeue above) will cancel your job whether it's running or pending.

Interactive job

To get connected to a compute node (i.e., interactive mode/shell), request a compute node from the slurm scheduler: 

interact -A cmda3634_rjh -p dev_q -t 00:30:00
The arguments after interact are

  • -A cmda3634_rjh tells which allocation to use
  • -p dev_q tells which queue to use (the "develop" queue is much shorter than a "normal" queue)
  • -t 00:30:00 the time to keep the interactive job running, in the format Hours:Minutes:Seconds

References