Runtime Environment

Parallel Environment
MPI
OpenMP

Parallel Environment

The Parallel Environment (PE) from IBM is designed for the development and execution of parallel C, C++, and Fortran programs and has the following:

• Parallel Operating Environment for submitting and managing jobs;
• Message passing libraries (MPI and MPL);
• Parallel debugger pdbx; and
• Xprofiler for analyzing a parallel application's performance.

Parallel Operating Environment

The Parallel Operating Environment (POE) is one part of the Parallel Environment. It can be looked upon as a "user interface" to PE and has an interactive parallel shell with a syntax similar to ksh.

Setting the Environment Variables

There are two ways to configure the way a parallel program is executed -- with environment variables or command-line arguments. The POE environment variables are set using either setenv (csh/tcsh) or export (ksh, bsh) commands and can be set:

• at the command prompt;
• in the "dot" files (.cshrc_user, .profile_user); or
• in the job script file.

The environment variables can be over-ridden with POE command line flags. The environment variable names are the same as the command line option names, but they start with "MP_" and are all in upper case. For example, the command line flag -procs has the corresponding environment variable "MP_PROCS." The following are a few, typical POE command line flags:

Set number of nodes or tasks

• procs - number of task processes
• nodes - number of physical nodes on which to run parallel tasks
• tasks_per_node - number of tasks to be run on each of the physical nodes

Those interested in using POE command line flags exclusively, instead of using them through job scripts, should learn about the remaining flags viewed here. Those interested in using the environment variables through the use of Loadleveler job scripts, should go here.

MPI Parallel Programming Environment

The IBM Power5 nodes are connected to each other via the high performance singly-linked Federation or High Performance Switch (HPS) interconnect. All off node communication is available for use through the IP protocol as well as the recommended User Space (US) protocol for better performance over the HPS switch using both switch interfaces or adaptors on each node (device is "sn_all"). Users can thus run their MPI or OpenMP applications with the HPS switch interconnect with either protocol submitted through their batch scripts. In addition, the MPI implementation now uses threads instead of signals for asynchronization activities.

MPI can use several protocols for communication between tasks. IP (internet protocol) can be used for communication between tasks on a node or on different nodes. Aside from relatively high latencies, this also incurs high overhead from use of the slow IP protocol. This availability is a useful alternative on such systems which do not have the SP interconnect switch, since it is the only mode of communication protocol for ethernet, GigE networks etc., for MPI tasks. Off node communication between tasks is fastest when the US (user space) protocol is used over the HPS. Although use of this protocol significantly reduces the overhead and latency when compared to IP, on node communication may be more efficiently handled using the shared memory protocol.

In addition, the MPI implementation is now layered over the Low-level Application Programming Interface (LAPI) transport layer , now available in the current version of the PE. In contrast to the "reliable byte stream" approach of the previous PE, LAPI provides a "reliable message" protocol, which uses much less storage for jobs with a large number of tasks. It is based on an "active message style" mechanism that provides a one-sided communications model in which one process initiates an operation and the completion of that operation does not require any other process to take a complementary action. In addition, the 64-bit MPI library has shown improved performance particularly with features such as RDMA, so it is recommended for users to build there application in 64-bit, even if they do not have any large memory requirements.

The Remote Direct Memory Access (RDMA) data transfer feature has also been made available on the High Performance Switch interconnect. RDMA allows the network adapter direct access to the user memory of the application.This allows the network adapter to move data between computational tasks with minimal CPU involvement, offering greater potential for overlapping calculation and communication when using non-blocking communication. RDMA enables large messages to utilise both network adapters attached to each of the compute nodes of the p5-575 system without utilising multiple CPUs. This promises a substantially increased bandwidth for large messages, particularly for the 64-bit applications. We show below how RDMA use is enabled by environment variables.

The parallel operating environment or POE is responsible, among other things, for managing all communication between the MPI processes whether on-node or between nodes. The runtime behavior of MPI jobs can be modified by configuring the many environment variables in POE.

OpenMP Parallel Programming Environment

Implementation of shared memory parallelization is done through the creation of user threads, which are mapped to kernel threads by the operating system.

OpenMP codes run initially with just one user thread, but a team of threads is created when the first parallel region is encountered. Upon exiting the parallel region, only one thread resumes execution of the serial portion, while the rest consumes CPU time waiting for the next parallel section. The maximum time a thread spends waiting is regulated by the environment variable SPINLOOPTIME. After this maximum interval is exceeded, a thread can either go to sleep or yield its place on the kernel to another runnable thread, provided that a yield time has been specified as part of the OpenMP environment. Reactivation of a sleeping thread is more costly than one that is in a yielded state.

The behavior of threads can be modified by setting several POE environment variables. Please check the reference for more details.

Running Interactive Programs

All parallel jobs run under the Parallel Operating Environment (POE). Use the interactive session only to develop and debug your parallel programs. Use a command similar to the one given below for interactive jobs.

% poe your_job job_args -tasks_per_node n -nodes 1 -rmpool 1

n is the number of processors on which one wants to run. Interactive runs, which are queued through the development class, are limited to up to 5 processors, for 4 hours and to upto 8 GBytes of total memory requirements.

Running Batch Code using LoadLeveler

What is LoadLeveler?
LoadLeveler Batch Resources
Building a Simple Job Command File
Building an Advanced Job Command File
LoadLeveler Keywords
Submitting a Job
Managing a Job
Summary of LoadLeveler Commands

What is the LoadLeveler?

LoadLeveler is a batch job scheduling software that provides the ability to submit and manage jobs. It has three interfaces.

• Command line interface - provides access to basic job and administrative functions.
• Graphical user interface (xloadl) - provides the same functionality as the command line interface; however, the command line interface is generally found to be more efficient.
• Application programming interface (api) - allows application programs that are written by users and administrators to interact with the LoadLeveler environment.

LoadLeveler Batch Resources on Champion

New Resource Limits

Following the recent upgrade with the IBM Power5 Node(s), the entire system consists of the eleven eight-way IBM Power5 HPC nodes, and a login node with 7 processors available for computation. The complete system consists of 96 Power5 compute processors with an aggregate memory of 192 GB, 7.2 TB of total disk and theoretical peak performance of about 730 Gigaflops .

In order to optimize node usage or job throughput on the newly configured Power5 system, a new resource policy has been implemented. The following table summarizes the new resource limits:

Table I: New resource limits on the TACC IBM Power5 system

# cpus	memory	walltime	class
2-5 procs	1.8 GB/procs	4 hrs limit	development
1-2 procs	1.8 GB/procs	12 hrs limit	serial
upto 32 procs	1.8 GB/procs	24 hrs. limit	normal

Users need to understand the logic employed by the filter which parses the jobscript commands and schedules the job accordingly. Due to the limitation of the total processor counts, and the homogeneous nature of the Champion system, relative to the Longhorn system, the filter logic is simplified. If certain user options are not allowed, then the filter provides the closest options for the user to make the appropriate choices. The filter logic employs the following general logic guidelines:

• User need to specify if they require node usage to be shared or not, unless all eight processors of the node(s) are in use. Then the node usage automatically becomes "not shared" or dedicated. The filter will return the job to the user if this requirement is not fulfillled in the script.
• All jobs requiring usage of a single node or greater, will require dedicated usage. In this situation, users are requested to submit processor counts which are multiples of eight, if feasible. If the user requests, for example 20 processors or 2.5 nodes, they will be charged for using three nodes.
• In case of users requiring processor counts less than eight, their first option is to use the development queue. In case of non-availability or if the necessary run time or the memory requirements exceeds that offered in the development queue, then users should require four processor count in normal queue for shared usage. Due to limited resources and for fair useage, users cannot run on processor counts different from four for sharing usage on a single node. They would need to resubmit the job with the correct processor count of four.
• Application codes using threads for OpenMP code or a hybrid MPI OpenMPI code would be allocated usage of a single processor for use of each thread. However, users must make sure that the total number of processors and threads that are in use, does not exceed the total number of "(processors) x (nodes)".
• Interactive jobs are currently queued through the development queue class only, and not allowed to run through any other available queue class. Similarly, interactive serial jobs are queued through the serial queue class. This policy will be enforced for fairness to all users. Users who need special requests such as benchmarks, which fall beyond the perview of the current class queues should make these requests well ahead of their scheduled runs.

Building a Simple Job Command File

For all batch submitted jobs, a job script or job command file needs to be created. This section is for the early or first time users to the IBM system who need to start with a simple job script described in this section. Users already experienced with Loadleveler job scripts should go to the next section where the intricacies for building more advanced and detailed job scripts are described.

Some of the general rules and information necessary for building a simple job script are given below followed by examples of scripts for an MPI and OpenMP job.

1. Keyword statements, which are case insensitive, begin with #@ followed by LoadLeveler keywords and statement components; to comment out any statements begin with ##;
2. Common functionality such as shell, initialdir, input, output, job name etc., are common to all job scripts, and should be taken verbatim from the examples below and modified according to specific user and job.
3. Shell command statements can go anywhere except where LoadLeveler keyword statements are being defined.
4. A special keyword statement used in the simple job script is the total number of tasks for MPI jobs and number of threads for OpenMP jobs. Either of these options should be specified by the user in the job script as tasks or threads , depending on whether it is an MPI or an OpenMP job respectively.
5. Users should also have an idea of the amount of memory the job consumes. If it is not specified, the job filter assumes the maximum limit which may not be sufficient for the job. The job will then be scheduled and may exit with erroneous statements. The memory statement is another special keyword statement and can be specified as

   #@ memory = 2

   This implies that the memory per task or thread is 2MB.The total memory is then computed by the job filter by multiplying it with the number of tasks or threads that are specified. Note that generally memory per task or thread should not exceed 2 GB or 2000 MB.

6. Finally, the user should have a rough idea of the runtime. This is for fairness of how jobs are scheduled. Jobs which take a shorter runtime are scheduled ahead of those that are longer, all other things being equal, so the user may be penalised in seeing the expected turnaround time. The specification for the runtime, another special keyword statement, can be done as follows:

   #@ walltime = 00:10:00

   This specifies that the wall clock time is 10 minutes.

7. Based on the user providing correct information on all of the above three special keyword statements, the job filter will then infer the other keyword statements that are necessary for the job to be scheduled. In certain situations, these generated conditions may create contradictory options for the filter, which is then returned to the user for making the appropriate choice.
8. Another Loadleveler keyword statement

   # @ job_type = parallel

   also has to be added in addition to the special keyword statements above for all MPI and OpenMP jobs. Although this keyword statement is not a special one, it is necessary for the job filter to know the classification of the job.

9. Users who belong to multiple projects should specify the project name to charge, as shown below for the A-abc project:

   # @ account_no = A-abc

   Project names are listed in the accounting report at login.

10. The environment variable OMP_NUM_THREADS has to be specified by the user for OpenMP jobs. All other environment variables will be automatically generated by the filter for both OpenMP and MPI jobs.

Note: Users should be aware that all of the special keyword statements can ONLY be used in the simple job scripts. Any combination of the special keyword statements with some of the more advanced keyword statements will result in incorrect filtering and will result in either the script being returned to the user or an erroneous job script sent to the scheduler.

An example of a job command file for MPI job follows:

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ notify_user = your_email #@ job_type = parallel #@ environment = COPY_ALL; #@ walltime = 01:00:00 #@ tasks = 8 #@ memory = 100 #@ class = normal #@ queue poe ./a.out

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ notify_user = your_email #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ job_type = parallel #@ environment = COPY_ALL; LL_JOB=TRUE; #@ walltime = 01:00:00 #@ threads = 8 #@ memory = 200 #@ class = normal #@ queue setenv OMP_NUM_THREADS 8 ./a.out

Building an Advanced Job Command File

Every LoadLeveler job must be submitted through a job command file. The job command file has LoadLeveler statement lines with LoadLeveler keywords that describe the job. Some of the general rules/information are explained below, in addition to explaining the more important and complicated LoadLeveler keyword statements whose understanding is absolutely essential.

Note that the special keywords used for the simple jobscript CANNOT be used for the advanced version of the job script described in this section.

Different MPI and OpenMP jobs require separate job scripts and some examples of the different scenarios are listed following the explanation of the general rules below.

1. Keyword statements, which are case insensitive, begin with #@ followed by LoadLeveler keywords and statement components; to comment out any statements begin with ##;
2. Shell command statements can go anywhere except where LoadLeveler keyword statements are being defined.

3. The fine grained resource management on node is done by the Work Load Manager (WLM). The following resource statement must be included in batch scripts.

   #@ resources = ConsumableCpus(CPUSpTask) ConsumableMemory (MEMpTASK)

            where  MEMpTASK = amount of MEMory  required Per TASK,
            and   CPUSpTASK = number of CPUs    required Per TASK.
            For OpenMP jobs use the value of OMP_NUM_THREADS for CPUSpTASK.
            For MPI    jobs use the value of        1        for CPUSpTASK.
           

   E.g. if ConsumableMemory(1024MB) is used with tasks_per_node=4, then 4096 Megabytes of memory will be reserved for the job. For OpenMP programs there is only 1 task, so you must make MEMpTASK equal to the amount of memory you need for ALL threads. [A master thread and its spawned group of threads is a single task and the ConsumableCPU's argument is the number of CPUs reserved per (for the) task.]

4. Parallel MPI jobs that run across multiple nodes have to specify the use of the HPS Switch interconnect for off-node communication. This involves setting MP_EUILIB to US and MP_EUIDEVICE to sn_all. A concise way of doing this is to set the #@ network.MPI_LAPI keyword to the following

   #@ network.MPI_LAPI = <device>,<shared|not_shared>,<protocol>

   where <device> =	sn_all	uses dual-plane Switch adapters
   	en1	Gigabit ethernet adapter
   where <protocol> =	US	to be used by the dual-plane Switch adaptors only
   	IP	can be used by ALL of the interconnects

5. A number of environment variables , when used appropriately impact performance. Some of the environment variables are set as default by the system. Other environment variables need to be used under certain conditions. Here are a listing of some of them:

   Default Settings

   These settings are already set by default and the users need not declare them in the jobscript, unless they need to be changed.

   	 MP_SHARED_MEMORY=yes 
   	
   	 MP_EUIDEVICE=sn_all  
   
   	 MEMORY_AFFINITY=MCM 
   

   Using RDMA

   Users wanting to use the RDMA feature need to set the following environment in there job scripts, when running there 64-bit application

   	  MP_USE_BULK_XFER=yes  
   
   	  MP_BULK_MIN_MSG_SIZE=150K, (Note: this size can be increased upto 2MB.)   
   
           

   Special Environment Use

   The following environments can be declared and used in most of the general conditions but not under all circumstances. These exceptions are pointed out.

   	 MP_SINGLE_THREAD=yes (Except for  MPI-I/O or explict MPI threading routines)  
   
   	 MP_TASK_AFFINITY=MCM (Except OpenMP routines) 
   

6. All OpenMP jobs, whether "serial" or "parallel", and hybrid MPI-OpenMP jobs which are "parallel", are so termed simply from the point of the number of tasks that are executed. But each of these tasks spawns multiple threads per process or task. This is done by the

   #@ resources = ConsumableCpus(n)..

   statement, where n is an integer between 1-8 for p575 HPC nodes. But this spawning of threads is equivalent to reserving n processes, not setting OpenMP threads. This is done by the statement

   setenv OMP_NUM_THREADS n

7. For all MPI, OpenMP or a hybrid combination jobs, the job filter evaluates the number of processors to reserve per node by the following formula

   number of processors/node = tasks_per_node x ConsumableCpus(n)

   For MPI jobs, n=1, while for OpenMP jobs, n=OMP_NUM_THREADS.

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ job_type = parallel #@ environment = COPY_ALL; MP_SINGLE_THREAD=yes; #@ resources = ConsumableCpus(1) ConsumableMemory(100mb) #@ network.MPI_LAPI = sn_all, not_shared, US #@ wall_clock_limit = 01:00:00 #@ node = 1 #@ tasks_per_node = 8 #@ node_usage = not_shared #@ notify_user = your_email #@ notification = error #@ class = normal #@ queue poe ./a.out

An example of a job command file for MPI jobs on multiple nodes on p575 nodes , follows:

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ job_type = parallel #@ environment = COPY_ALL; MP_SINGLE_THREAD=yes; #@ node_usage = not_shared #@ resources = ConsumableCpus(1) ConsumableMemory(100mb) #@ wall_clock_limit = 24:00:00 #@ network.MPI_LAPI = sn_all, not_shared, US #@ node = 4 #@ tasks_per_node = 8 #@ notification = never #@ class = normal #@ queue poe ./a.out

Note: Comparing the two MPI job scripts for running application codes on one node and for more than node, the main difference is specifying the appropriate network/adaptor information. In the latter script, that is contained in the line

#@ network.MPI_LAPI = sn_all, not_shared, US

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ job_type = parallel #@ environment = COPY_ALL; LL_JOB=TRUE; #@ resources = ConsumableCpus(8) ConsumableMemory(1000mb) #@ network.MPI_LAPI = sn_all, not_shared, US #@ wall_clock_limit = 01:00:00 #@ node = 1 #@ tasks_per_node = 1 #@ node_usage = not_shared #@ notify_user = your_email #@ notification = error #@ class = normal #@ queue setenv OMP_NUM_THREADS 8 ./a.out

Note that unlike the MPI jobs, the ConsumableCpus(8) statement above is critical for OpenMP jobs, which is "serial" in terms of processors (tasks_per_node = 1), but uses multiple threads per process or task. This statement reserves 8 processes, but does not set the number of OpenMP threads which is always done by the environment variable OMP_NUM_THREADS . See Rule 5 for further explanation.

An example of a job command file for hybrid MPI-OpenMP job across multiple p690 HPC nodes, follows: (Note that here a single MPI process uses multiple OpenMP threads.)

#@ shell = /bin/csh #@ initialdir = /home/login_name/work_dir #@ job_name = my_job #@ input = /dev/null #@ output = $(job_name).o$(jobid) #@ error = $(job_name).o$(jobid) #@ job_type = parallel #@ environment = COPY_ALL; LL_JOB=TRUE #@ resources = ConsumableCpus(4) ConsumableMemory(1000mb) #@ wall_clock_limit = 24:00:00 #@ node = 2 #@ tasks_per_node = 2 #@ network.MPI_LAPI = sn_all,not_shared,US #@ node_usage = not_shared #@ notification = error #@ class = normal #@ queue setenv OMP_NUM_THREADS 4 poe ./a.out

Note that in the above the job script, the #@ resources = ConsumableCpus(4) statement is critical for OpenMP jobs. In the above script, 4 total processors are reserved for MPI tasks (tasks_per_node x node) and 4 threads are reserved for each process or task (ConsumableCpus). Also note that this statement reserves 8 processes, but does not set the number of OpenMP threads which is done by the environment variable OMP_NUM_THREADS , which must ALWAYS be used for setting the correct number of OpenMP threads. See Rule 6 for additional information.

LoadLeveler Keywords

Keyword	Description	Example
environment	specifies the inital environment variables when the job starts -- environment specifications should be separated with semicolons	COPY_ALL specifies all environment variables from your shell be copied var = value specifies the environment variable to which var should be set
error	specifies the name of the file to use as standard error when the job step runs	error = file_name
executable	for serial jobs, executable identifies the name of the program to run -- if this keyword is not included and the job command file is a shell script, LoadLeveler uses the script file as the executable	executable = script
initialdir	this is the pathname to the working directory during execution of the job step	initialdir = pathname or "." for current working dir.
input	specifies the name of the file to use as standard input when the job step is run	input = file_name (/dev/null if no input is reqd.)
job_name	specifies the name of the job -- the name can be any combination of letters, numbers, or both	job_name = my_job
job_type	specifies the type of job to process -- valid entries are "parallel" or "serial"	job_type = string
node	specifies the minimum and maximum number of nodes requested by a job step -- at least one of these values must be specified	node = [min][,max]
notification	secifies when the user named in the notify_user keyword is sent mail	notification = always | error | start | never | complete always - notifies the user when the job begins, ends, or if there is an error error - notifies the user only if the job fails start - notifies the user only when the job begins never - never notifies the user complete - notifies the user when the job ends -- this is the default
notify_user	specifies the user to whom mail is sent based on the notification keyword - the submitting user and the submitting machine are the defaults	notify_user = login_name@tacc.utexas.edu
output	specifies the name of the file to use as standard output when a job step runs	output = file_name
node_usage	whether a user wants to share their memory in the SMP node with other users. Depending on the class of job i.e. on 4 processors or 8 processors, the job filter always changes it to shared.	node_usage = shared or not_shared
queue	places one copy of the job step in the queue -- this statement is required and essentially marks the end of the job step
resources	specifies quantities of the consumable resources "consumed" by each job. The resources may be machine resources or floating resources.	resources = name(count) ... name(count) where name could be ConsumableCpus or ConsumableMemory. The count for ConsumableCpus is an integer between 1-8. The count for ConsumableMemory is integral units expressed in terms of MB or GB. The user requirement is checked against a filter and the job is not submitted if memory requirements are exceeded.
shell	specifies the name of the shell to use for the job step -- the default is the shell used in the owner's password file entry	shell = name
network device	specifies the correct interconnect device for use; The other devices is en1	MP_EUIDEVICE = sn_all; #@ network.MPI = csss;..;..
network internode communication	specifies that parallel MPI programs use fastest internode communication available; other option is using ip protocol	MP_EUILIB = us #@ network.MPI = ..;..;us
tasks_per_node	specifies the number of tasks to run per node of a parallel job and should be used in conjunction with the node keyword -- the maximum number of tasks a job step can request is limited by the total_tasks keyword in the administration file	tasks_per_node = number
wall_clock_limit	user limit for the elapsed time for job, but is tested for hard-limit by the filter	wall_clock_limit = 04:00:00

Submitting a Job

A job can be submitted using the GUI xloadl or at the command line with the following code.

llsubmit my_job

LoadLeveler assigns the job a three part identifier and also sets the environment variables for the job. The identifier consists of the following:

• Machine name: the name of the machine that schedules the job and is not necessarily the name of the machine that runs the job;
• Job ID: an identifier that is assigned to a group of jobs that are initiated from the same job command file; and
• Step (or Process) ID: an identifier that is unique for every job and begins at zero and increments by one for each job that is in the same job command file.

Managing a Job We will go through some of the basic Loadleveler commands to illustrate how they can be used to monitor and/or manage user jobs.

Showing the Status of a Job

The LoadLeveler command llq is used to obtain information about jobs in the LoadLeveler queue. The following is a small snippet of job listing that is obtained with the llq command showing all three of the components of the heterogenous Power4 system. These listings are headed by abbreviated and/or cryptic acronyms (in some cases), which we will then explain.

Id                     Owner      Submitted  ST  PRI Class        Running On
------------          ----------- ---------  --  --- -----------  ----------
champ01.440.0          ux454432    7/13 11:23 R  50  normal       champ10
champ01.447.0          mlane       7/16 16:22 R  50  normal       champ07

Id:	The identification number of the user submitted Job. Use this number to check status or cancel the job.
ST:	State of job;   R (running), job is currently running.   I (Idle), job is waiting in queue to run but not currently running.   H (hold), job is placed on hold by the user. The job will not run unless released from Hold state
Class:	Which class the job is running. User determines if it is either normal or development and then the appropriate class according to the nodes is determined by the filter.
Running On:	The node(s) -- champ10, champ07 or champ11 where the job has started running. The other numbers in the sequence are only of importance for system administration purposes and should not be of concern to the user.

Note: For special cases including when users erroneously access the $ARCHIVE filesystem at runtime, the job hangs in an idle state rather than quit with appropriate error messages. After the job tries to run once, the "(alloc)" state is displayed in the "Running on" column in the llq output.

Placing and releasing a hold on a job

A job can be placed on hold, where it will remain in the queue until it is released. To place a hold on a job with the identifier machine.123.0 use the following code.

llhold machine.123.0

To release the hold on the above job

llhold -r machine.123.0

Displaying the Status of a Machine

To display information about the machines in the LoadLeveler pool use the llstatus command. For a long listing use the -l flag as follows.

llstatus -l

Cancelling a Job

The llcancel command can be used to cancel either running or queued jobs as follows.

llcancel machine.123.0

Summary of LoadLeveler Commands

Command	Function
llcancel	cancels a submitted job
llclass	returns information about LoadLeveler classes
llckpt	used to checkpoint a single job step
llextSDR	extracts adapter information from the system data repository (SDR)
llhold	holds or releases a hold on a job
llmatrix	returns GANG matrix information in the LoadLeveler cluster when GANG scheduling is used
llmodify	changes attributes or characteristics of a submitted job step
llprio	changes the user priority
llq	queries the status of LoadLeveler jobs
llstatus	queries the status of LoadLeveler machines
llsubmit	submits a job
llsummary	returns resource information on completed jobs

Program Timers & Performance Tools

Measuring the performance of a program should be an integral part of code development. It provides benchmarks to gauge the effectiveness of performance modifications and can be used to evaluate the scalability of the whole package and/or specific routines. There are quite a few tools for measuring performance, ranging from simple timers to hardware counters. Reporting methods vary too, from simple ASCII text to X-Window graphs of time series.

The most accurate way to evaluate changes in overall performance is to measure the wall-clock (real) time when an executable is running in a dedicated environment. On Symmetric Multi-Processor (SMP) machines, where resources are shared (e.g., the TACC IBM Power4 P690 nodes), user time plus sys time is a reasonable metric; but the values will not be as consistent as when running without any other user processes on the system. The user and sys times are the amount of time a user's application executes the code's instructions and the amount of time the kernel spends executing system calls on behalf of the user, respectively.

Package Timers

The time command is available on most Unix systems. In some shells there is a built-in time command, but it doesn't have the functionality of the command found in /usr/bin. Therefore you might have to use the full pathname to access the time command in /usr/bin. To measure a program's time, run the executable with time using the syntax "/usr/bin/time -p <args>" (-p specifies traditional "precision" output, units in seconds):

/usr/bin/time -p ./a.out	{Time for a.out execution}
real 1.54	{Output (in seconds)}
user 0.5
sys 0
/usr/bin/time -p mpirun -np 4 ./a.out	{Time for rank 0 task}

The MPI example above only gives the timing information for the rank 0 task on the master node (the node that executes the job script); however, the real time is applicable to all tasks since MPI tasks terminate together. The user and sys times may vary markedly from task to task if they do not perform the same amount of computational work (are not load balanced).

Code Section Timers

"Section" timing is another popular mechanism for obtaining timing information. The performance of individual routines or blocks of code can be measured with section timers by inserting the timer calls before and after the regions of interest. Several of the more common timers and their characteristics are listed below in Table 1.

Table 1. Code Section Timers
Routine	Type	Resolution (usec)	OS/Compiler
times	user/sys	1000	Linux/AIX/IRIX/UNICOS
getrusage	wall/user/sys	1000	Linux/AIX/IRIX
gettimeofday	wall clock	1	Linux/AIX/IRIX/UNICOS
rdtsc	wall clock	0.1	Linux
read_real_time	wall clock	0.001	AIX
system_clock	wall clock	system dependent	Fortran 90 Intrinsic
MPI_Wtime	wall clock	system dependent	MPI Library (C & Fortran)

For general purpose or course-grain timings, precision is not important; therefore, the millisecond and MPI/Fortran timers should be sufficient. These timers are available on many systems; and hence, can also be used when portability is important. For benchmarking loops, it is best to use the most accurate timer (and time as many loop iterations as possible to obtain a time duration of at least an order of magnitude larger than the timer resolution). The times, getrussage, gettimeofday, rdtsc, and read_real_time timers have been packaged into a group of C wrapper routines (also callable from Fortran). The routines are function calls that return double (precision) floating point numbers with units in seconds. All of these TACC wrapper timers (x_timer) can be accesses in the same way:

     external   x_timer                 double x_timer(void);
     real*8  :: x_timer                 ...
     real*8  :: sec0, sec1, tseconds    double sec0, sec1, tseconds;
     ...                                ...
     sec0     = x_timer()               sec0     = x_timer();
     ...Fortran Code                    ...C Codes
     sec1     = x_timer()               sec1     = x_timer();
     tseconds = sec1-sec0               tseconds = sec1-sec0

The wrappers and a makefile are available with a test example from TACC for instrumenting codes.

Standard Profilers

The gprof profiling tool provides a convenient mechanism to obtain timing information for an entire program or package. Gprof reports a basic profile of how much time is spent in each subroutine and can direct developers to where optimization might be beneficial to the most time-consuming routines, the "hotspots". As with all profiling tools, the code must be instrumented to collect the timing data and then executed to create a raw-date report file. Finally, the data file must be read and translated into an ASCII report or a graphic display. The instrumentation is accomplished by simply recompiling the code using the -qp (Intel compiler) option. The compilation, execution, and profiler commands for gprof are shown below for a Fortran program:

Profiling Serial Executables

ifc -qp prog.f90	{Instruments code}
a.out	{Produces gmon.out trace file}
gprof	{Reads gmon.out (default args: a.out gmon.out) (report sent to STDOUT)}

Profiling Parallel Executables

mpif90 -qp prog.f90	{Instruments code}
setenv GMON_OUT_PREFIX gout.*	{Forces each tasks to produce a gout.<pid>}
mpirun -np <#> a.out	{Produces gmon.out trace file}
gprof -s gout.*	{Combines gout files into gmon.sum}
gprof a.out gmon.sum	{Reads executable (a.out) & gmon.sum (report sent to STDOUT)}

Detailed documentation is available at www.gnu.org. A documented example of a gprof flat profile and call graph output is available to help you interpret gprof output.

Timing Tools

Most of the advanced timing tools access hardware counters and can provide performance characteristics about floating point/integer operations, as well as memory access, cache misses/hits, and instruction counts. Some tools can provide statistics for an entire executable with little or no instrumentation, while others requires source code modification.

PAPI Toolkit

The PAPI (Performance Application Programming Interface) toolkit is available on Tejas and Lonestar. A user guide and overview are available at the University of Tennessee-Knoxville Innovative Computing Laboratory's PAPI home page. PAPI collects hardware monitor data and provides raw counts and/or derived performance information. A simple easy-to-use, high-level interface is available for most of the common performance metrics. There is also a low-level interface for more control and direct access to the native hardware events.

To use PAPI on TACC systems, first load the PAPI module using the command module load papi. This will set some environment variables that can then be used for compiling and linking. Once the PAPI module is loaded, a list of the high-level PAPI counters available on the system can be shown by running $PAPI_DIR/ctests/avail. The third column of the output indicates if the PAPI counter is available for use on the system processor. The available low-level native counters can be listed by running $PAPI_DIR/ctests/native_avail. The "$PAPI_DIR/ctests" and "$PAPI_DIR/ftests" directories contain many programs which can be examined to see examples of how PAPI may be implemented. Some of the more interesting events collected by PAPI are shown in Table 2 below. It is important to note that there are a limited number of counters available and some PAPI event counters may interfere with other counters. It is up to the user to ensure that the desired counters are operating correctly. Also, make sure to always check the error codes to verify that the PAPI functions are running properly. An example C program that can be used to test the different counters can be downloaded from here.

IMPORTANT: Programs with PAPI functions will not run correctly on the Lonestar head node. Only run these programs by submitting them as batch jobs on the compute nodes.

Table 2. Example PAPI Events
PAPI Name	Description
PAPI_BR_INS	Branch Instructions
PAPI_FP_INS	Floating Point Instructions
PAPI_TOT_INS	Total Instructions completed
PAPI_L2_DCM	L2 Data Cache Misses
PAPI_L2_LDM	L2 Load Misses
PAPI_TLB_DM	Data TLB Misses
PAPI_TLB_TM	Total TLB Misses

The simplest method for instrumentating your progam with PAPI is to use the high level function PAPI_flops in C or PAPIF_flops for Fortran. This function is called once at the beginning to initialize the PAPI counters and timers, then each subsequent call returns the total wall time, processor time, and floating point operations since the initial PAPI_flops call. An example C program using PAPI_flops is shown below

   #include <papi.h>
   #include <stdio.h>

   int handle_error(int code)
   {
      char errmesg[PAPI_MAX_STR_LEN];
      PAPI_perror(code, errmesg, PAPI_MAX_STR_LEN);
      printf("PAPI Error %i: %s\n", code, errmesg);
   }

   int main()
   {
     int err_code;
     float wall_time, proc_time, mflops;
     long_long floatpnt_ops;

     if ((err_code=PAPI_flops(&wall_time, &proc_time, &floatpnt_ops, &mflops)) != PAPI_OK)
         handle_error(err_code);

     ...<main code>...

     if ((err_code=PAPI_flops(&wall_time, &proc_time, &floatpnt_ops, &mflops)) != PAPI_OK)
         handle_error(err_code);

     printf("Wall time = %g\n", wall_time);
     printf("Processor time = %g\n", proc_time);
     printf("MFLOPS = %g\n", floatpnt_ops/proc_time);
    }

Note: the MegaFLOPS value returned by PAPI_flops function in PAPI 3.0 Beta may be incorrect, hence, the actual value is determined in the above code by taking the number of floating point operations and dividing by the processor time. Compiling the above code is accomplished by issuing the command:

The PAPI instrumented example Fortran code below uses the high-level PAPI counter functions to collect the number of floating point operations, total processor cycles, total number of instructions issued and the number of L2 data cache misses.

     program example
     #include "f90papi.h"
     integer, parameter :: nevents = 4
     integer*4 ::  ievents(nevents), ierr
     integer*8 ::  icounts(nevents)
     ...
     events(1)= PAPI_FP_OPS   !  This event is for PAPI 3.0 and later
     events(2)= PAPI_TOT_CYC
     events(3)= PAPI_TOT_INS
     events(4)= PAPI_L2_DCM


     call PAPIF_start_counters(ievents,nevents,ierr)
     ...<main code>...
     call PAPIF_read_counters( icounts,nevents,ierr)
     print*,icounts
     stop
     end

PAPI's default loader mode is to dynamically load its library objects into executables; therefore, it is best to use the -rpath loader option to indicate the location of the libraries at run time. If you loaded the papi module, the library path is automatically included in LD_LIBRARY_PATH, and the library and include paths are set in PAPI_LIB and PAPI_INC, respectively. Compiling a Fortran code instrumented with PAPI has the following syntax:

Further details and examples can be found on the PAPI home page.

Tuning an application for single-processor performance is important for two reasons. First, optimized code runs faster and provides quicker turnaround, thereby delivering results back to the user sooner. Secondly, code optimization has a positive impact on system throughput and facilitates the processing of more jobs in a given time period; it will benefit your colleagues. Therefore, programmers should be conscientious about improving code performance for the benefit of the programmer/user and to prevent inefficient code (resource-hogging) from lowering system throughput, thus inconveniencing other users waiting for free computational resources.

For an in-depth look at single processor optimization on the IBM Power4 System click here.

Most programs that perform scientific and technical computation depend on common mathematical operations (e.g. matrix-vector multiplication, dot products) and/or spend a considerable amount of time calling mathematical functions like sqrt, sin, log, etc. For this purpose, IBM has developed several high performance libraries specifically optimized for the Power4 architecture. Users are encouraged to link in these mathematical and scientific libraries instead of hand-coding their own routines:

1. ESSL library

   The Engineering and Scientific Subroutine Library (ESSL) consists of routines in the following computational areas:

   • BLAS (vector-vector, matrix-vector, matrix-matrix operations)
   • linear algebraic equations
   • eigensystem analysis
   • Fourier transforms, convolutions and correlations, and other signal processing operations
   • sorting and searching
   • interpolation
   • numerical quadrature
   • random number generation

   Two types of runtime libraries are provided by ESSL:

   • The ESSL SMP (symmetric multiprocessing) library provides thread-safe versions of the ESSL routines for use on SMP architectures. Some subroutines are even designed for multithreaded execution. Consult this table for a list of these routines that support shared memory parallel processing.

     To call an ESSL routine with multithreaded programming support (i.e. routines that can execute in parallel using multiple threads), link in the ESSL SMP library, e.g.:

     xlf90_r -qsmp=auto prog.f -lessl_r

     Parallel execution of ESSL can only take place when the number of threads specified in the runtime environment is greater than one. To set the number of threads, adjust the OMP_NUM_THREADS environment variable:

     setenv OMP_NUM_THREADS n (in csh)

     export OMP_NUM_THREADS=n (in sh)

     Please consult chapter 7 of the POWER4 Processor Introduction and Tuning Guide for a discussion of the environment variables that affect runtime behavior of SMP applications.

   • The ESSL serial library provides thread-safe versions of the ESSL subroutines for use on all processors. Serial applications can link in this library using:

     xlf90 prog.f -lessl

   Refer to the ESSL User's Guide for more information on the use of this library, as well as details on the calling interface and syntax of each routine.
2. PESSL library

   Parallel ESSL is the scalable version of ESSL, and uses the message passing (MPI) library for multiprocessor execution of mathematical computational routines in the following areas:

   • PBLAS levels 2 and 3 (matrix-vector and matrix-matrix operations)
   • linear algebraic equations
   • eigensystem analysis and SVD
   • Fourier transforms
   • random number generation

   Because PESSL routines make calls to MPI, use the "mp" invocation of the compiler when compiling and linking in the PESSl library, e.g.:

   mpxlf90 prog.f -lpessl mpcc prog.c -lpessl

   On the Power4 system, PESSL SMP libraries are provided for use with the MPI thread-safe library. Note that PESSL routines cannot be called simultaneously from multiple threads -- that is, calls to PESSL must not be within an SMP parallel region. Use the SMP version of PESSL when linking code with the thread-safe MPI libraries:

   mpxlf90_r prog.f -lpessl_r

   Both SMP and "serial" versions of PESSl support 64-bit and 32-bit applications.

   For more information on the PESSL library, please refer to the PESSL User's Guide.
3. MASS and MASSV Libraries

   Short for Mathematical Acceleration Subsystem Library, MASS provides fast, high-performance versions of a subset of Fortran intrinsic functions at the expense of decreased accuracy (i.e. the last one or two bits might be inaccurate).

   For vector operands, the MASSV library should be used instead. However, this would require modification of the code to call the vector equivalent of the intrinsic function. For example:

   Integer, Parameter :: n=1000 real(8) a(n) ... do i=1,n ... a(i)=sqrt(1.0/a(i)) ... end do

   should be replaced by:

   ... call vsrsqrt(a,a,n)    ! 32-bit environment call vrsqrt(a,a,n)    ! 64-bit environment ...

   Execute the following command when linking in the MASS or MASSV libraries, making sure to link in MASS before libm.a:

   xlc prog.c -L/usr/local/mass -lmass -lm xlc prog.c -L/usr/local/mass -lmassvp4 -lm

   Fortran codes link in libm.a automatically so this need not be linked in explicitly. However, use -lmass prior to -lxlf90 in the load step.

   MASS contains support for both 64-bit and 32-bit applications, except for vatan2 and vsatan2.

   Please consult /usr/local/apps/MASS.readme or this page for more information on the scalar and vector versions of the MASS library.

• IBM Documentation for XLF 10.1 and XLC 8.0 for AIX
• pSeries and AIX 5.3 Information Center
• Help on GPFS 2.3, ESSL 4.2, PESSL 3.2, PE 4.2.2, Loadleveler 3.3.1
• SSH Website
• TACC Introduction to SSH
• TACC Detailed Review of SSH
• "High Performance Computing" by Kevin Dowd and Charles Severance (O'Reilly book)