Ranger User Guide

Login Shell

The most important component of a user's environment is the login shell that interprets text on each interactive command line and statements in shell scripts. Each login has a line entry in the /etc/passwd file, and the last field contains the shell launched at login. To determine your login shell, execute:

echo $SHELL

{to see your login shell}

You can use the chsh command to change your login shell; instructions are in the man page. Available shells are listed in the /etc/shells file with their full-path. To change your login shell, execute:

cat /etc/shells	{select a from list}
chsh -s	{use full path of the shell}

---

User Environment

The next most important component of a user's environment is the set of environment variables. Many of the Unix commands and tools, such as the compilers, debuggers, profilers, editors, and just about all applications that have GUIs (Graphical User Interfaces), look in the environment for variables that specify information they may need to access. To see the variables in your environment execute the command:

env {to see environment variables}

The variables are listed as keyword/value pairs separated by an equal (=) sign, as illustrated below by the HOME and PATH variables.

HOME=/home/utexas/staff/milfeld

PATH=/bin:/usr/bin:/usr/local/apps:/opt/intel/bin

(PATH has a colon (:) separated list of paths for its value.) It is important to realize that variables set in the environment (with setenv for C shells and export for Bourne shells) are "carried" to the environment of shell scripts and new shell invocations, while normal "shell" variables (created with the set command) are useful only in the present shell. Only environment variables are seen in the env (or printenv) command; execute set to see the (normal) shell variables.

---

Startup Scripts

All Unix systems set up a default environment and provide administrators and users with the ability to execute additional Unix commands to alter the environment. These commands are "sourced"; that is, they are executed by your login shell, and the variables (both normal and environmental) as well as aliases and functions are included in the present environment. We recommend that you customize the login environment by inserting your "startup" commands in .cshrc_user, .login_user, and .profile_user files in your home directory.

Basic site environment variables and aliases are set in

/usr/local/etc/cshrc	{C-shell, non-login specific}
/usr/local/etc/login	{C-shell, specific to login}
/usr/local/etc/profile	{Bourne-type shells}

For historical reasons, the C shells source two types of files. The .cshrc type files are sourced first (/etc/csh.cshrc--> $HOME/.cshrc--> /usr/local/etc/cshrc--> $HOME/.cshrc_user). These files are used to set up environments that are to be executed by all scripts and used for access to the machine without a login. For example, the following commands only execute the .cshrc type files on the remote machine:

scp data ranger.tacc.utexas.edu:	{only .cshrc sourced on ranger}
ssh ranger.tacc.utexas.edu date	{only .cshrc sourced on ranger}

The .login type files are used to setup environment variables that you commonly use in an interactive session. They are sourced after the .cshrc type files (/etc/csh.login--> $HOME/.login--> /usr/local/etc/login-->
$HOME/.login_user). Similarly, if your login shell is a Bourne shell (bash, sh, ksh, ...), the profile files are sourced (/etc/profile--> $HOME/.profile--> /usr/local/etc/profile--> $HOME/.profile_user).

The commands in the /etc files above are concerned with operating system behavior and set the initial PATH, ulimit, umask, and environment variables such as the HOSTNAME. They also source command scripts in /etc/profile.d -- the /etc/csh.cshrc sources files ending in .csh, and /etc/profile sources files ending in .sh. Many site administrators use these scripts to setup the environments for common user tools (vim, less, etc.) and system utilities (ganglia, modules, Globus, LSF, etc.)

TACC has to coordinate the environments on platforms of several operating systems: AIX, Linux, IRIX, Solaris, and Unicos. In order to efficiently maintain and create a common environment among these systems, TACC uses its own startup files in /usr/local/etc. (A corresponding file in this etc directory is sourced by the .profile, , and .login files that reside in your home directory. (Please do not remove these files and the sourcing commands in them, even if you are a Unix guru.) Any commands that you put in your .login_user, .cshrc_user, or .profile_user file are sourced (if the file exists) at the end of the corresponding /usr/local/etc command files. If you accidentally remove your .login, .cshrc, and .login, you can copy new ones from /usr/local/etc/start-up or execute

/usr/local/bin/install_ut_startups

to get a new copy (your old files are renamed with a date suffix).

---

Modules

TACC is constantly including updates and installing revisions for application packages, compilers, communications libraries, and tools and math libraries. To facilitate the task of updating and to provide a uniform mechanism for accessing different revisions of software, TACC uses the modules utility.

At login, a basic environment for the default compilers, tools, and libraries is set by several modules commands. Your PATH, MANPATH, LIBPATH, directory locations (WORK, HOME, ...), aliases (cdw ...) and license paths, are just a few of the environment variables and aliases created for you. This frees you from having to initially set them and update them whenever modifications and updates are made in system and application software.

Users who need 3rd party applications, special libraries, and tools for their development can quickly tailor their environment with only the applications and tools they need. (Building your own specific application environment through modules allows you to keep your environment free from the clutter of all the other application environments you don't need.)

Each of the major TACC applications has a modulefile that sets, unsets, appends to, or prepends to environment variables such as $PATH, $LD_LIBRARY_PATH, $INCLUDE_PATH, $MANPATH for the specific application. Each modulefile also sets functions or aliases for use with the application. A user need only invoke a single command,

module load

at each login to configure an application/programming environment properly. If you often need an application environment, place the modules command in your .login_user and/or .profile_user shell startup file.

Most of the package directories are in /opt/apps ($APPS) and are named after the package name (). In each package directory there are subdirectories that contain the specific versions of the package.

As an example, the fftw package requires several environment variables that point to its home, libraries, include files, and documentation. These can be set in your environment by loading the fftw module:

module load fftw

To see a list of available modules and a synopsis of a modulefile's operations, execute:

module available	{lists modules}
module help	{lists environment changes performed for }

During upgrades, new modulefiles are created to reflect the changes made to the environment variables. TACC will always announce upgrades and module changes in advance.

home directory

At login, the system automatically changes to your home directory.
This is the recommended location to store your source codes and build your executables.
The quota limit on home is 6GB.
A user's home directory is accessible from the frontend node and any compute node.
Use $HOME to reference your home directory in scripts.
Use cd to change to $HOME.

work directory

Store large files here.
Often users change to this directory in their batch scripts and run their jobs in this file system.
A user's work directory is accessible from the frontend node and any compute node.
The user quota is 350GB for Early Users, and will be set to a smaller value when $SCRATCH comes on line.
Purge Policy: There is no purging on this file system.
This file system is not backed up.
Use $WORK to reference your work directory in scripts.
Use cdw to change to $WORK.

scratch or temporary directory

UNLIKE the TACC Lonestar system, this is NOT a local disk file system on each node.
This is a global Lustre file system for storing temporary files.
Purge Policy: Files on this system are purged when a file's access time exceeds 10 days.
PLEASE NOTE: TACC staff may delete files from scratch if the scratch file system becomes full and directories consume an inordinately large amount of disk space, even if files are less than 10 days old. A full file system inhibits use of the file system for all users. The use of programs or scripts to actively circumvent the file purge policy will not be tolerated.
Often, in batch jobs it is more efficient to use and store files directly in $WORK (to avoid moving files from scratch later before they are purged).
The quota on this system is 400TB.
Use $SCRATCH to reference this file system in scripts.
Use cds to change to $SCRATCH.

archive

More on Archive -- rls and sinc not available yet

High Speed File Transfers

There are two utilities, bbcopy and globus-url-copy, that can be used to achieve higher performance than the rcp and scp programs, when transferring large files between TACC clusters and the TACC archive (ranch). During production, the scp and rcp speeds between Ranger and Ranch average about 15MB/s, while bbcopy and globus-url-copy speeds are about 125MB/s. These values vary with I/O and network traffic.

The bbcopy utility works much the same way as scp, it is only available on TACC machines and includes an option to copy subdirectories. For each transfer command it is necessary to provide your ssh passphrase or login password. You can use the ssh-agent and ssh-add commands to automatically supply passphrases for ssh commands (including bbcp) during your login session. Details are available in the ssh Cookbook section of the TACC SSH Guide. The bbcp syntax is:

bbcp   -help     {displays option list}
bbcp   options   <file or directory>   <toMachine>:<relativepath>/<fileordirectory>

e.g.

bbcp data ranch:	{tranfers "data" to ranch as "data"}
bbcp -f data ${ARCHIVER}:data	{transfers "data" to ranch, force replacement}
bbcp -r dir1 ${ARCHIVER}:	{transfers directory dir1, and subdirectories}

By default, files are not overwritten, use the -f to force replacement of files. The -r option recursively transfers subdirectories. You can also use bbcp to transfer files between lonestar and ranger.

TeraGrid users who wish to transfer data between TG sites can use globus-url-copy. This command requires the use of your TeraGrid certificate to create a proxy for passwordless transfers. It is syntactically complex, but provides high-speed access to other TeraGrid machines that support gridFTP services (protocol for globus-url-copy). High-speed transfers are accomplished by tranferring file/directories between the different FTP servers at the TG sites. The file systems of the compute machines are mounted on GridFTP servers, thereby providing the access to your files/directories through these servers. Third party transfers are possible (transfer initiated between two machines from another machine). All TG GridFTP servers and mounted directory names are listed in the TG transfer-location page.

It is necessary to use grid-proxy-init to obtain a proxy certificate. The proxy is valid for 12 hours for all logins on the local machine. The startup commands are:

grid-proxy-init

{prompted for certificate password, proxy good for 12hrs.}

With globus-url-copy you must include the name of the server, and a full path to the file. The syntax is:

globus-url-copy <options>	<gsiftp://<gridftp_server1>/<directory>/file> \
	<gsiftp://<gridftp_server2>/<directory>/file>

e.g. file transfer:
globus-url-copy -stripe -tcp-bs 11M -vb \
gsiftp://gridftp.ranger.tacc.teragrid.org/`pwd`/file1 \
gsiftp://tg-gridftp.ncsa.teragrid.org/home/ncsa/johndoe/file2
e.g. directory transfer:
globus-url-copy -stripe -tcp-bs 11M -vb \
gsiftp://gridftp.ranger.tacc.teragrid.org/`pwd`/directory1/ \
gsiftp://gridftp.ranch.tacc.teragrid.org/home/00000/johndoe/directory2/

It is important to the use the stripe and buffer size options (-stripe to use multiple service nodes, -tcp-bs 11M to set ftp data channel buffer size), otherwise the speed will be about 20 times slower! By default all paths are root-relative-- that is, a full path must be specified. Note: when transferring directories, the directory path MUST end with a slash (/). The -rp option (not shown) allows paths relative to the user's "starting" directory of the filesystem mounted on the server.

Runtime Environment

Bindings to the most recent shared libraries are configured in the file /etc/ld.so.conf (and cached in the /etc/ld.so.cache file). Cat /etc/ld.so.conf to see the TACC configured directories, or execute

/sbin/ldconfig -p

to see a list of directories and candidate libraries. Use the -Wl,rpath loader option or the LD_LIBARY_PATH to override the default runtime bindings.

The Intel compiler and MKL math libraries are located in the /opt/intel directory (installation date TBD), and application libraries are located in /usr/local/apps ($APPS). The GOTO libraries are located in /opt/apps/gotoblas/gotoblas-1.02 (installation date TBD). Use the

module help libname

command to display instructions and examples on loading libraries.

---

The SGE Batch System

Batch facilities like LoadLeveler, NQS, LSF, OpenPBS or SGE differ in their user interface as well as implementation of the batch environment. Common to all, however, is the availability of tools and commands to perform the most important operations in batch processing: job submission, job monitoring, and job control (hold, delete, resource request modification). In Section I basic batch operations and their options are described. Section II discusses the SGE batch environment, and Section III provides the queue structure on SGE. In the references at the end of this section there are links to SGE manuals. New user should visit the SGE wiki and read the first chapter of the Introduction to the N1 Grid Engine 6 Software document.. To help users who are migrating from other systems, a comparison of the IBM LoadLeveler, OpenPBS, LSF and SGE batch options and commands is presented in a separate document.

Section I: Three Operations of Batch Processing: submission, monitoring, and control

Step 1: Job submission

SGE provides the qsub command for submitting batch jobs: Use the SGE qsub command to submit a batch job with the following syntax:

qsub job_script

where job_script is the name of a file with unix commands. This "job script" file can contain both shell commands and special commented statements that include qsub options and resource specifications. Details on how to build a script follow.

Table 1. List of the Most Common qsub Options
Option	Argument	Function
-q	queue_name	Submits to queue designated by queue_name
-pe	pe_name min_proc[-max_proc]	Executes job via the Parallel Environemnt designated by pe_name with min_proc-max_proc number of processes
-N	job_name	Names the job job_name
-S	shell (absolute path)	Use shell as shell for the batch session
-M	emailaddress	Specify user's email address
-m	{b|e|a|s|n}	Specify when user notifications are to be sent
-V		Use current environment setting in batch job
-cwd		Use current directory as the job's working directory
-o	output_file	Direct job output to output_file
-e	error_file	Direct job error to error_file
-A	account_name	Charges run to account_name. Used only for multi-project logins. Account names and reports are displayed at login.
-l	resource=value	Specify resource limits (see qsub man page)

Options can be passed to qsub on the command line or, specified in the job script file. The latter approach is preferable. It is easier to store commonly used qsub commands in a script file that will be reused several times rather than retyping the qsub commands at every batch request. In addition, it is easier to maintain a consistent batch environment across runs if the same options are stored in a reusable job script.

Batch scripts contain two types of statements: special comments and shell commands. Special comment lines begin with #$ and are followed with qsub options. The SGE shell_start_mode has been set to unix_behavior which means that the Unix shell commands are interpreted by the shell specified on the first line after #! sentinel; otherwise the Bourne shell (/bin/sh) is used. The file job below requests an MPI job with 32 cores and 1.5 hours of run time:

#!/bin/bash               
#$ -V                     # Inherit the submission environment
#$ -cwd                   # Start job in  submission directory
#$ -N myMPI               # Job Name
#$ -j y                   # combine stderr & stdout into stdout    
#$ -o $JOB_NAME.o$JOB_ID  # Name of the output file (eg. myMPI.oJobID)
#$ -pe 16way 32           # Requests 16 cores/node, 32 cores total
#$ -q normal              # Queue name
#$ -l h_rt=01:30:00       # Run time (hh:mm:ss) - 1.5 hours
## -M myEmailAddress      # Email notification address (UNCOMMENT)
## -m be                  # Email at Begin/End of job  (UNCOMMENT)
 set -x                   #{echo cmds, use "set echo" in csh}
 ibrun ./a.out            # Run the MPI executable named "a.out"

If you don't want stderr and stdout directed to the same file, remove don't include a -j option line, and insert a -e option (stderr) to name the file that is to receive stderr output.

MPI Environment for Scalable Code

The MVAPICH-1 and MVAPICH-2(default) MPI packages provide runtime environments that can be tuned for scalable code. For packages with short messages, there is a "FAST_PATH" option that can reduce communication costs, as well as a mechanism to "Share Receive Queues". Also, there is a "Hot-Spot Congestion Avoidance" option for quelling communication patterns that produce hot spots in the switch. See Chapter 9, "Scalable features for Large Scale Clusters and Performance Tuning" and Chapter 10, "MVAPICH2 Parameters" of the MVAPICH2 User Guide for more information. The User Guides are available in PDF format at:

MVAPICH User Guides

Understanding the SGE Parallel Environment

Each Ranger node (of 16 cores) can only be assigned to one user; hence, a complete node is dedicated to a user's job and accrues wall-clock time for 16 cores whether they are used or not. The SGE parallel environment option "-pe" sets the number of MPI Tasks per Node (TpN), and the Number of Nodes (NoN). The syntax is:

#$ -pe way
e.g.
#$ -pe 16way 64 {16 MPI tasks per node, 4 nodes (= a total of 64 assigned cores/16)}

where:
TpN is the Task per Node, and
NoN is the Number of Nodes requested.

Note, regardless of the value of TpN, the second argument is always 16 times the number of nodes that you are requesting.

Using a multiple of 16 cores per node

For "pure" MPI applications, the most cost-efficient choices are: 16 tasks per node (16way) and a total number of tasks that is a multiple of 16. This will ensure that each core on all the nodes is assigned one task. In this case use:

$# -pe 16way

Using fewer than 16 cores per node

When you want to use less that 16 MPI tasks per node, the choice of tasks per node is limited to the set of numbers {1, 2, 4, 8, 12, and 15}. When the number of tasks you need is equal to "Number of Tasks per Node x Number of Nodes", then use the following prescription:

$# -pe way

where
TpN is a number in the set {1, 2, 4, 8, 12, 15}

If the total number of tasks that you need is less than "Number of Tasks per Node x Number of Nodes", then set the MY_NSLOTS environment variable to the total number of tasks. In a job script, use the following -pe option and environment variable statement:

$# -pe way
...
setenv MY_NSLOTS { C-type shells }
or
export MY_NSLOTS= { Bourne-type shells }

e.g.
$# -pe & 8way 64 {use 8 Tasks per Node, 4 Nodes requested}
...
setenv MY_NSLOTS 31 {31 tasks are launched}

where
TpN is a number in the set {1, 2, 4, 8, 12, 15}

Program Environment for Hybrid Programs

For hybrid jobs, specify the MPI Tasks per Node through the first -pe option (1/2/4/8/15/16way) and the Number of Nodes in the second -pe argument (as the number of assigned cores = Number of Nodes x 16). Then, use the OMP_NUM_THREADS environment variable to set the number of threads per task. (Make sure that "Tasks per Node x number of Nodes" is less than or equal to the number assigned cores, the second argument of the -pe option.) The hybrid job script below illustrates the use of these parameters to run a hybrid job. It requests 4 tasks per node, 4 threads per task, and a total of 32 cores (2 nodes x 16 cores).

#!/bin/bash
#	{use bash shell}
...
#$ -pe 4way 32
#	{4 cores/node, 32 cores total}
...
set -x	#{echo cmds, use "set echo" in csh}
setenv OMP_NUM_THREADS 4
	#{4 threads/task}
ibrun ./hybrid.exe

The job output and error are sent to out.o and err.o, respectively. SGE provides several environment variables for the #$ options lines that are evaluated at submission time. The above $JOB_ID string is substituted with the job id. The job name (set with -N) is assigned to the environment variable JOB_NAME. The memory limit per task on a node is automatically adjusted to the maximum memory available to a user application (for serial and parallel codes).

Step 2: Batch query

After job submission, users can monitor the status of their jobs with the qstat command. Table 2 lists the qstat options:

Table 2. List of qstats Options
Option	Result
-t	Show additional information about subtasks
-r	Show resource requirements of jobs
-ext	Displays extended information about jobs
-j	Displays information for specified job
-qs {a|c|d|o|s|u|A|C|D|E|S}	Show jobs in the specified state(s)
-f	Shows "full" list of queue/job details

The qstat command output includes a listing of jobs and the following fields for each job:

Table 3. Some of the fields in qstats command output
Field	Description
JOBID	job id assigned to the job
USER	user who owns the job
STATE	current job status, includes (but not limited to)
w	waiting
s	suspended
r	running
j	on hold
E	errored
d	deleted

For convenience, TACC has created an additional job monitoring utility which summarizes all jobs in the batch system in a manner similar to the "showq" utility from PBS. Execute

showq

to summarize all running, idle, and pending jobs, along with any advanced reservations scheduled within the next week. Note that showq -u will show jobs associated with your userid only (issue showq --help to obtain more information on available options). An example output from showq is shown below:

ACTIVE JOBS--------------------  

JOBID   JOBNAME USERNAME        STATE   PROC    REMAINING       STARTTIME
14694   equillda        user1   Running 16      18:54:07  Tue Feb 3 17:32:41
14701          V    user2  Running    16    7:02:41  Tue Feb 3 17:41:15  
14707          V    user3  Running    16   19:11:02  Tue Feb 3 17:49:36  
14708      jet08    user4  Running    32    0:38:36  Tue Feb 3 18:17:10  
14713        rti    user5  Running    64    3:58:25  Tue Feb 3 20:36:59  
14714        cyl    user6  Running   128   23:16:36  Tue Feb 3 21:55:10          

6 Active jobs     272 of 556 Processors Active (48.92%)

IDLE JOBS---------------------- 
JOBID  JOBNAME   USERNAME    STATE  PROC    WCLIMIT       QUEUETIME    
14716     bigjob    user7     Idle   512    0:15:00  Tue Feb 3 22:18:57  
14719   smalljob    user7     Idle   256    0:15:00  Tue Feb 3 22:35:31           

2 Idle jobs    

BLOCKED JOBS-------------------  
JOBID  JOBNAME   USERNAME    STATE  PROC    WCLIMIT       QUEUETIME    
14717      hello    user7     Held    16    0:15:00  Tue Feb 3 22:19:07  
14718      hello    user7     Held    32    0:15:00  Tue Feb 3 22:19:15           

4 Blocked jobs    Total Jobs: 12    Active Jobs: 6     Idle Jobs: 2     Blocked Jobs: 4    

ADVANCED RESERVATIONS----------  
RESV ID   PROC                    RESERVATION WINDOW  
karl#79    556    Tue Mar 23 09:00:00 2004 - Tue Mar 23 17:30:00 2004

The latest queue information can be determined from the following commands (a single command to extract queue structure information will be available later):

qconf sql {Lists the available queues.}
qconf sq { s_rt/h_rt are the soft/hard wall-clock limits}
cat /share/sge/default/tacc/sge_esub_control {first value after max_cores_per_job_ is the queue core limit}

Step 3: Job control

Control of job behavior takes many forms:

1. Job modification while in the pending/run state
   Users can reset the qsub options of a pending jo