<![CDATA[
<body>
<html>
<head>
<title>Champion User Guide</title>
<script type="text/javascript" src="/styles/stylesheet.php"></script>
</head>
<body bgcolor=#013166 leftmargin=8 topmargin=8 marginwidth=8 marginheight=8 valign=top >
<table align=center valign=top border=0 cellpadding=0 cellspacing=0 width=849>
<tr>
<td nowrap bgcolor="#013166" align=center valign=middle colspan=2>
<img src="/images/champion_header.gif" border=0></td>
</tr>
<tr>
<td colspan=2>
<table border=0>
<tr>
<td valign="top" width="125px"><table bgcolor="#013166" align=left valign=top border=0 cellpadding=0 cellspacing=0>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="spacer image"
height="1px" width="130px"></td>
</tr>
<tr>
<td>
<table align=left valign=middle border=0 cellpadding=0 cellspacing=0
bgcolor="#013166" width="130px">
<tr align=left valign=middle bgcolor="#013166">
<td nowrap valign=middle width="115px" bgcolor="#013166"
class="nav">General Sys. Info</td>
<td valign=middle bgcolor="#013166" width="15px">
<img src="/images/blue_spacer.gif" alt="15 pixel spacer
image">
</td>
</tr>
</table>
</td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="" height="1px"
width="130px"></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#architecture" class="nav2" >Architecture</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#access" class="nav2" >System Access</a></td>
</tr>
</table></td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="spacer image"
height="1px" width="130px"></td>
</tr>
<tr>
<td>
<table align=left valign=middle border=0 cellpadding=0 cellspacing=0
bgcolor="#013166" width="130px">
<tr align=left valign=middle bgcolor="#013166">
<td nowrap valign=middle width="115px" bgcolor="#013166"
class="nav">Environment</td>
<td valign=middle bgcolor="#013166" width="15px">
<img src="/images/blue_spacer.gif" alt="15 pixel spacer
image">
</td>
</tr>
</table>
</td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="" height="1px"
width="130px"></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#login" class="nav2" >Login Info</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#filesystems" class="nav2" >File Systems</a></td>
</tr>
</table></td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="spacer image"
height="1px" width="130px"></td>
</tr>
<tr>
<td>
<table align=left valign=middle border=0 cellpadding=0 cellspacing=0
bgcolor="#013166" width="130px">
<tr align=left valign=middle bgcolor="#013166">
<td nowrap valign=middle width="115px" bgcolor="#013166"
class="nav">Development</td>
<td valign=middle bgcolor="#013166" width="15px">
<img src="/images/blue_spacer.gif" alt="15 pixel spacer
image">
</td>
</tr>
</table>
</td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="" height="1px"
width="130px"></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#models" class="nav2" >Prog. Models</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#compilation" class="nav2" >Compilation</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#running" class="nav2" >Running Code</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#tools" class="nav2" >Tools</a></td>
</tr>
</table></td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="spacer image"
height="1px" width="130px"></td>
</tr>
<tr>
<td>
<table align=left valign=middle border=0 cellpadding=0 cellspacing=0
bgcolor="#013166" width="130px">
<tr align=left valign=middle bgcolor="#013166">
<td nowrap valign=middle width="115px" bgcolor="#013166"
class="nav">Optimization</td>
<td valign=middle bgcolor="#013166" width="15px">
<img src="/images/blue_spacer.gif" alt="15 pixel spacer
image">
</td>
</tr>
</table>
</td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="" height="1px"
width="130px"></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#single" class="nav2" >Single Processor</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#libraries" class="nav2" >Perf. Libraries</a></td>
</tr>
</table></td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="spacer image"
height="1px" width="130px"></td>
</tr>
<tr>
<td>
<table align=left valign=middle border=0 cellpadding=0 cellspacing=0
bgcolor="#013166" width="130px">
<tr align=left valign=middle bgcolor="#013166">
<td nowrap valign=middle width="115px" bgcolor="#013166"
class="nav">References</td>
<td valign=middle bgcolor="#013166" width="15px">
<img src="/images/blue_spacer.gif" alt="15 pixel spacer
image">
</td>
</tr>
</table>
</td>
</tr>
<tr bgcolor="#C32736">
<td style="font-size:1px; line-height:1px;" bgColor="#C32736">
<img src="/images/red_spacer.gif" alt="" height="1px"
width="130px"></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="noframes.php?sys=champion#manuals" class="nav2" >Manuals</a></td>
</tr>
</table></td>
</tr>
<tr>
<td>
<table bgcolor="#013166" align=left valign=top width="130px"
border=0 cellpadding=0 cellspacing=2>
<tr align=left valign=top>
<td align=left valign=middle height="10px">
<img src="/images/blue_small_spacer.gif"
width="20px" height="10px"></td>
<td nowrap align=left valign=middle width="110px"
height="10px">
<a href="/resources/software" class="nav2" target="_top">Packages</a></td>
</tr>
</table></td>
</tr>
</table></td>
<td background="/images/watermark.jpg">
<style type="text/css">
<!--
.style1 {color: #FF0000}
-->
</style>
<!-- <p class="style1"><strong>IMPORTANT:</strong> <strong>All users are affected by the migration from Power4 to Power5. </strong><a href="Pwr5migration1.pdf">Please first read the Migration Guide</a>.</p> -->
<p> <a name="architecture"></a>
</p>
<table border=0 cellpadding=0 cellspacing=0 align=center height=60px>
<tr>
<td align=center valign=middle class="title">Architecture</td>
</tr>
</table>
<div class="text">
TACC's enhanced IBM Power5 System, Champion, now has a homogeneous selection of
resources for both MPI and OMP parallel computing paradigms. It also
provides almost 50% improvement for peak performance on a per processor basis,
and improved sustained performance compared to the previous heterogeneous
configuration of Longhorn.
<p>
Champion consists of 12 nodes each with 8
processors, aggregating to a total of 96 processors. Ten of the twelve
nodes are used for dedicated jobs, or where the node usage is not shared.
In the remaining two nodes, including the login or frontend node, node usage
is shared between multiple concurrent jobs.
One of the 8 processors in the frontend node will be used for
the General Parallel File System (GPFS). Two frontend
proccessors will be used for compiling. A visual of the single p5-575 frame
with configuration details is illustrated below in Figure 1.
<!-- <<<<<< architecture.php
<table><tr><td><img src="/services/userguides/champion/champ1-2.jpg"></td></tr><tr><td><center> Figure 1. Complete Power5 System Configuration </center> </td></tr></table>
======= -->
<table width="100%"><tr><td align="center"><img src="/services/userguides/champion/champ1-2.jpg"></td></tr><tr><td align="center">Figure 1. Power5 System Configuration </td></tr></table>
<p>
All the nodes employ the same 1.9GHz IBM Power5 microarchitecture, run
the same AIX 5.3 operating system, and are connected through an
IBM High Performance "Federation" Switch (HPS). These features provide
a common and integrated parallel programming environment throughout
the system. The homogeneous node-architectures provide different
computational spaces for parallel algorithms and paradigms.
<p>
The architectural features of the Power5 chip are designed with a high
memory bandwidth to accommodate the superscalar operations of a 1.9
Gigahertz processor. Features such as speculative branching,
out-of-order execution, predication, 8-stream prefetching, and a
3-tier cache hierarchy provide a continuous and high data throughput
for high peak performance.
<p>
<!--
The basic building blocks of IBM's p5-575 HPC system node are eight
Dual-Chip Modules or DCMs. These are packaged, unlike the Power4 MCMs,
in a more traditional chip packaging fashion. The DCM consists of the Power5
chip (on a 389mm**2 die) and an L3-cache chip. Each Power5 chip is
dual-core, with only one core or CPU active on TACC's Champion
system (see Figure 2). This allows for greater bandwidth for the single active
microprocessor across all levels of the memory hierarchy, and thus is
extremely beneficial for most HPC applications. Each chip also
contains the fabric and memory controller and is also sometimes referred
to as "system on a chip".
-->
<!-- <<<<<< architecture.php
<table><tr><td><center><img src="/services/userguides/champion/pwr5-chip2.jpg"></center></td></tr><tr><td><center>Figure 2. Power5 Chip </center> </td></tr></table>
======= -->
<table width="100%"><tr><td align="center"><img src="/services/userguides/champion/pwr5-chip2.jpg"></td></tr><tr><td align="center">Figure 2. Power5 Chip </td></tr></table>
<p>
<!-- img("3","Power4 Memory and I/O Path, and Chip-to-Chip Crossbar Components"); ?> -->
Some important design changes were made to the Power5
chip that improved the overall system performance. One such change is
moving the L3 cache from the memory side of the ASIC fabric to the
processor side of the fabric, but off-chip. This removed the L3
cache from the path between the chip and the memory controller, as was
the case in Power4. This allowed for the memory controller to be moved
from the fabric to on-chip. This can be seen in Figure 2.
These two changes also had significant
additional benefits: reduced latency to the L3 and to memory,
increased bandwidth to the L3 cache and the memory, and improvement in
intrinsic reliability resulting from the reduction in the number of
chips necessary to build a system. The p5-575 system thus
comprises of two chips: a POWER5 chip and a L3
cache chip. Finally both the General-Purpose Registers and the
Floating-Point registers have been increased to 120, from 80 and 72
respectively, in the Power4 chip. The additional FP rename capability
had the benefit of increasing Power5's single thread performance on
some HPC workloads. This gain stemmed from an enhanced ability to
execute critical code sections out of order.
<p> The cache and memory
characteristics for Power5 are as follows:
<ul>
<li>L1 cache consists of 32 KB data with write-through, 4-way set associativity with a cache line size of 128 bytes. The transfer rate from L1 data cache to registers is about 2 words/clock period or about 30 GB/s with a latency of 2 clock periods. The L1 instruction is 64 KB and is direct-mapped.</li>
<p>
<li>L2 cache is 1.9 MB with write-in, 10-way set associativity with a cache line size of 128 bytes. The transfer rate from L2 to L1 data cache is 4 words/clock period or
about 60 GB/s with a latency of 10 clock periods. </li>
<p>
<li>L3 cache is 36 MB in size with 12-way set associativity with a cache line size
of 256 bytes. The transfer rate from L3 to L2 cache is .87 words/clock period or about
9 GB/s with a latency of about 90 clock periods.</li> <p>
<li> The memory size consists of 2 GB per DCM (see below) or 16 GB per node aggregating to a
total of 192 GB for the whole system. The memory bandwidth is about 16 GB/s/chip with a latency of about 200 clock periods. This is about four times the bandwidth and a third less latency
when compared to the Power4 Longhorn system. From performance benchmarks the Power5 bandwidth
is about twice that observed from similar benchmarks in the similar Power4 systems.</li> <p>
</ul>
<!-- Because there are 4 L3 caches per mode, the memory-L3 cache size is effectively 4x128 bytes. The memory-to-L3 cache bandwidth is 13.9GB/sec.
-->
Each core processor has 2 floating point units that can each perform a
"fused" multiply-add operation per clock cycle. At a clock speed of
1.9GHz, four floating point operations per cycle can deliver 7.6 GFLOPS
from each processor. In all, the 96 processors of the TACC Power5
Champion system can deliver a peak performance of about 730 GFLOPS.
<p>
Beyond the die level, the next architectural level for the Power5 system
is the Dual-Chip Module (DCM). This is the basic building block of
IBM's p5-575 HPC system node which has 8 such DCMs.
Unlike the Power4 MCM which had 4 chip cores, there is only one processor chip
in a Power5 DCM and a L3 cache chip. Each processor chip is
dual-core, with only one core or CPU active on TACC's Champion
system (see Figure 2). This allows for greater bandwidth for the single active
microprocessor across all levels of the memory hierarchy, and thus is
extremely beneficial for most HPC applications.
<p>
Note that a considerable amount of the
silicon real estate for the Power5 chip is now devoted to the on-chip memory controller as
well as the directory for the L3 cache, and is consequently sometimes referred
to as a "system on a chip". These design changes and the
modest increases in L2 and L3 cache sizes as well as increased Simultaneous
Multi-Threading (SMT) functionality offered in the new Power5 chip,
have increased the Power5 die-size to 389 mm**2, from the previous 247 mm**2 size
for the Power4 chip.
<p>
<!--
the
a chip-to-chip interconnect
"controller" that interconnects to 3 other chips to form a distributed
crossbar switch. These interconnect fabric controllers allow each
Power4 chip to access the L3 cache and memory of the other chips.
Connecting 4 dies on a multichip module (Figure 4) forms an
"HPC" 4-way (single core) or a "Turbo" 8-way (dual core) SMP that
shares all L3 and memory modules associated with the MCM. The MCM in
Figure 4. shows the glass-ceramic layer of a 4-chip module. There are
over 5,000 module I/O pads that make connections to the board.
<img src="dcm3-2.jpg">
<center>
<p>
<img src="/images/pwr4_figure4.gif">
<p>
<font class="footer">Figure 4. IBM Power5 Dual-Chip Module (DCM)</font>
<p>
</center>
Within each die there is another distributed crossbar switch component that connects each chip's
intra-MCM switch to an equivalent chip on each of the other (possible) MCMs of a node (through a
logical connection to one of four unidirectional physical loops through the board). It is the
MCM-to-MCM interconnection that allows nodes to be built with multiple MCM modules (up to 4).
Figure 5. illustrates the MCM-MCM module interconnect for four MCMs, forming a fully populated
P690 frame (node). Our 32-way node consists of 4 Turbo MCMs (4 MCMs x 8 cores per module); our
16-way nodes are built with 4 HPC MCMs (4 MCMs x 4 cores per module); and our 4-way nodes consist
of a single HPC MCM (1 MCM x 4 cores per module). Each 16- and 32-way node is in a single frame,
while the 32 newer, 4-way nodes are compactly housed in only 2 frames.
<center>
<p>
<img src="/images/pwr4_figure5.gif">
<p>
<font class="footer">Figure 5. MCM-MCM Interconnect</font>
<p>
</center>
The 4-way nodes are ideal for MPI applications that require a large number of processors and
considerable interprocessor communication. Each node has two ports to the switch, and hence shares
the bandwidth among only 4 processors. On the other hand, the 16- and 32-way nodes, only have 2
ports for each node and must share off-node communication bandwidth with 16 and 32 processors,
respectively. Therefore, these nodes are more suited for small OMP applications requiring up to 32
processors and a large shared memory. Small MPI jobs, requiring 16 or less processors will also
run well on these nodes, provided the application runs solely on a single 16-way node and thereby
only uses on-node message passing. The SP Switch2 configuration diagram is shown in Figure 6. The
dual-plane configuration means that 2 ports (or connections) are used to double the bandwidth
between the nodes, while maintaining effectively two separate switches to minimize traffic
contention.
-->
<p>
<!-- <<<<<< architecture.php
<table><tr><td><img src="/services/userguides/champion/HPS-champ.jpg"></td></tr><tr><td><center> Figure 3. Champion Interconnect Configuration </center></td></tr></table>
======= -->
<table width="100%"><tr><td align="center"><img src="/services/userguides/champion/HPS-champ.jpg"></td></tr><tr><td align="center">Figure 3. Champion Interconnect Configuration </td></tr></table>
<p>
It is important to remember that there are two ports to each and every
node. However only one of the ports is linked to the High Performance
Switch as only one switch is currently needed for a system of this
size. This can be seen in Figure 3. The sustained switch performance should reflect this
configuration aspect. The peak observed MPI bandwidth
from the HPS switch is 1.88 GB/s (peak is 2 GB/s) while the reported latency is 4
us MPI task latency.
<!--Since the three
different node types have a different number of processors, the throughput (bandwidth) per
processor is different for each type of node. Hence, the 4-way P655 have the highest internode
bandwidth per processor, and the 16- and 32-way P690s have four and eight times less bandwidth,
respectively. (All on-node message passing occurs through shared memory, and should not differ as
much as the internode bandwidths-per-processor do among the node types.)
-->
<p>
TACC's Power5 system has a variety of disk solutions for different
storage needs. A General Parallel File System, GPFS, is available from
any node and provides parallel access to files through MPI-I/O or
native Unix calls with a distributed locking protocol for coherent
access from any node. This file system has 7.2 TB of disk and one GPFS
processor server, with large stripe groups across multiple disks. File
access is served over the High Performance "Federation" Switch for
high throughput. There are local scratch disks on each node, except
the login node, for tasks that do independent, local I/O. Home
directories are mounted on all computational nodes. Also, for
large-file, long term storage, the TACC archival file system is accessible
from the login node only. The Archive file system is not accessible
from any other Champion nodes outside of the frontend.
The connectivity and size of these file systems
are shown in Figure 7.
<!-- <center>
<p>
<img src="/images/pwr4_figure7.gif">
<p>
<font class="footer">Figure 7. Connectivity and Size of Power4 Filesytems</font>
<p>
</center> -->
</div>
Last modified: March 03 2008 11:01:59. <p>
<hr color="#013166" width="100%" size=2>
<p>
<!--Top of page Anchor-->
<a name="access"></a>
<table border=0 cellpadding=0 cellspacing=0 align=center height=60px>
<tr>
<td align=center avalign=middle>
<div class="title">System Access</div>
</td>
</tr>
</table>
<div class="text">
<p>
<font class="section">SSH</font>
<p>
To ensure a secure login session, users must connect to machines using the
secure shell, <b>ssh</b> program. <b>Telnet</b> is no longer allowed
because of the security vulnerabilities associated with it.
The <i>"r"</i> commands <b>rlogin, rsh</b>, and <b>rcp</b>, as well as <b>ftp</b>,
are also disabled on this machine for similar reasons. These commands are
replaced by the more secure alternatives included in SSH --- <b>ssh</b>,
<b>scp</b>, and <b>sftp</b>.
<P>
Before any login sessions can be initiated using <b>ssh</b>, a working
SSH client needs to be present in the local machine. Go to the <A
target="_top" href="http://www.tacc.utexas.edu/resources/userguides/ssh_intro"
class="link3">TACC introduction to SSH</a> for information on downloading
and installing SSH.
<P>
To initiate a ssh connection to a machine, type the following on the
local workstation
<BLOCKQUOTE>
<table border=0>
<tr>
<td nowrap><b><tt>ssh <login-name></tt></b>
<font class="text">@</font>
<tt><b><machine-name>.tacc.utexas.edu
</b></tt></td>
</tr>
</table>
</BLOCKQUOTE>
Note that the <login-name> is only needed if the user name on the machine being logged
onto differs from the user name on the workstation.
<p>
</div>
Last modified: October 04 2006 09:47:16. <p>
<hr color="#013166" width="100%" size=2>
<p>
<table align=center>
<tr height=60>
<td align=center valign=middle class=title><a name="login">Login Info</a></td>
</tr>
</table>
<table align=center>
<tr height=60>
<td nowrap><a href="#shell" class="link2">Login Shell</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#user" class="link2">User Environment</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#startup" class="link2">Startup Scripts</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#modules" class="link2">Modules</a></td>
</tr>
</table>
<p>
<div class="text">
<a name="shell"><font class=section>Login Shell</font></a>
<p>
The most important component of a user's environment is the login shell that interprets text on
each interactive command line and statements in shell scripts. Each login has a line entry in the
<b><code>/etc/passwd</code></b> file, and the last field contains the shell launched at login. To
determine your login shell, execute:
<p>
<center><code><b>grep <my_login_name> /etc/passwd</b></code> {to see your login shell}</center>
<p>
You can use the <code><b>chsh</b></code> command to change your login shell; instructions are in
the man page. Available shells are listed in the <code><b>/etc/shells</b></code> file with their
full-path. To change your login shell, execute:
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>cat /etc/shells</b></code></td>
<td class=text>{select a <shell> from list}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>chsh -s <shell> <username></b></code></td>
<td class=text>{use full path of the shell}</td>
</tr>
</table>
<p><hr color=#013166 size=2 width=50% align=center><p>
<a name=user><font class=section>User Environment</font></a>
<p>
The next most important component of a user's environment is the set of environment variables.
Many of the Unix commands and tools, such as the compilers, debuggers, profilers, editors, and
just about all applications that have GUIs (Graphical User Interfaces), look in the environment
for variables that specify information they may need to access. To see the variables in your
environment execute the command:
<p>
<center><code><b>env</b></code> {to see environment variables}</center>
<p>
The variables are listed as keyword/value pairs separated by an equal (=) sign, as illustrated
below by the HOME and PATH variables.
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>HOME=/home/utexas/staff/milfeld</b></code></td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>PATH=/bin:/usr/bin:/usr/local/apps:/opt/intel/bin</b></code></td>
</tr>
</table>
<p>
(PATH has a colon (:) separated list of paths for its value.) It is important to realize that
variables set in the environment (with <code><b>setenv</b></code> for C shells and
<code><b>export</b></code> for Bourne shells) are "carried" to the environment of shell scripts
and new shell invocations, while normal "shell" variables (created with the
<code><b>set</b></code> command) are useful only in the present shell. Only environment variables
are seen in the <code><b>env</b></code> (or <code><b>printenv</b></code>) command; execute <b>set</b> to
see the (normal) shell variables.
<p><hr color=#013166 size=2 width=50% align=center><p>
<a name=startup><font class=section>Startup Scripts</font></a>
<p>
All Unix systems set up a default environment and provide administrators and users with the
ability to execute additional Unix commands to alter the environment. These commands are
"sourced"; that is, they are executed by your login shell, and the variables (both normal and
environmental) as well as aliases and functions are included in the present environment. We
recommend that you customize the login environment by inserting your "startup" commands in
<code><b>.cshrc_user</b></code>, <code><b>.login_user</b></code>, and
<code><b>.profile_user</b></code> files in your home directory.
<p>
Basic site environment variables and aliases are set in
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>/etc/csh.cshrc</b></code></td>
<td class=text>{C-shell, non-login specific}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>/etc/csh.login</b></code></td>
<td class=text>{C-shell, specific to login}</td>
</tr>
<tr bgcolor=#e4e4e4>
<td><code><b>/etc/profile</b></code></td>
<td class=text>{Bourne-type shells}</td>
</tr>
</table>
<p>
For historical reasons, the C shells source two types of files. The <b><code>.cshrc</code></b>
type files are sourced first (<b><code>/etc/csh.cshrc--> $HOME/.cshrc--> /usr/local/cshrc-->
$HOME/.cshrc_user</code></b>). These files are used to set up environments that are to be executed
by all scripts and used for access to the machine without a login. For example, the following
commands only execute the <code><b>.cshrc</b></code> type files on the remote machine:
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>scp data lonestar.tacc.utexas.edu:</b></code></td>
<td class=text>{only .cshrc sourced on lonestar}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>ssh lonestar.tacc.utexas.edu date</b></code></td>
<td class=text>{only .cshrc sourced on lonestar}</td>
</tr>
</table>
<p>
The <code><b>.login</b></code> type files are used to setup environment variables that you
commonly use in an interactive session. They are sourced after the <code><b>.cshrc</b></code> type
files (<code><b>/etc/csh.login--> $HOME/.login--> /usr/local/login-->
<br>
$HOME/.login_user</b></code>). Similarly, if your login shell is a Bourne shell (bash, sh, ksh,
...), the profile files are sourced (<code><b>/etc/profile--> $HOME/.profile-->
/usr/local/profile--> $HOME/.profile_user</b></code>).
<p>
The commands in the <b><code>/etc</code></b> files above are concerned with operating system
behavior and set the initial <b><code>PATH</code></b>, <code><b>ulimit</b></code>,
<b><code>umask</code></b>, and environment variables such as the <code><b>HOSTNAME</b></code>.
They also source command scripts in <code><b>/etc/profile.d</b></code> -- the
<code><b>/etc/csh.cshrc</b></code> sources files ending in <code><b>.csh</b></code>, and
<b><code>/etc/profile</code></b> sources files ending in <code><b>.sh</b></code>. Many site
administrators use these scripts to setup the environments for common user tools (vim, less, etc.)
and system utilities (ganglia, modules, Globus, LSF, etc.)
<p>
TACC has to coordinate the environments on platforms of several operating systems: AIX, Linux,
IRIX, Solaris, and Unicos. In order to efficiently maintain and create a common environment among
these systems, TACC uses its own startup files in <b><code>/usr/local/etc</code></b>. (A
corresponding file in this <b><code>etc</b></code> directory is sourced by the
<b><code>.profile</code></b>, <b><code.cshrc</code></b>, and <b><code>.login</code></b> files that
reside in your home directory. (Please do not remove these files and the sourcing commands in
them, even if you are a Unix guru.) Any commands that you put in your
<b><code>.login_user</code></b>, <b><code>.cshrc_user</code></b>, or
<b><code>.profile_user</code></b> file are sourced (if the file exists) at the end of the corresponding
<b><code>/usr/local/etc</code></b> command files. If you accidentally remove your
<code><b>.login</b></code>, <code><b>.cshrc</b></code>, and <code><b>.login</b></code>, you can
copy new ones from <code><b>/usr/local/etc/start-up</b></code> or execute
<p>
<center><code><b>/usr/local/bin/install_ut_startups</b></code></center>
<p>
to get a new copy (your old files are renamed with a date suffix).
<p><hr color=#013166 size=2 width=50% align=center><p>
<a name=modules><font class=section>Modules</font></a>
<p>
TACC is constantly including updates and installing revisions for application packages, compilers,
communications libraries, and tools and math libraries. To facilitate the task of updating and to
provide a uniform mechanism for accessing different revisions of software, TACC uses the modules
utility.
<p>
At login, a basic environment for the default compilers, tools, and libraries is set by several
modules commands. Your <code><b>PATH</b></code>, <code><b>MANPATH</b></code>,
<code><b>LIBPATH</b></code>, directory locations (<code><b>WORK</b></code>,
<code><b>ARCHIVE</b></code>, <code><b>HOME</b></code>, ...), alias (<code><b>cdw</b></code>,
<code><b>cda</b></code>, ...) and license paths, are just a few of the environment variables and
aliases created for you. This frees you from having to initially set them and update them whenever
modifications and updates are made in system and application software.
<p>
Users who need 3rd party applications, special libraries, and tools for their development can
quickly tailor their environment with only the applications and tools they need. (Building your
own specific application environment through modules allows you to keep your environment free from
the clutter of all the other application environments you don't need.)
<p>
Each of the major TACC applications has a <b><code>modulefile</code></b> that sets, unsets,
appends to, or prepends to environment variables such as <code><b>$PATH</b></code>,
<code><b>$LD_LIBRARY_PATH</b></code>, <code><b>$INCLUDE_PATH</b></code>,
<code><b>$MANPATH</b></code> for the specific application. Each <code><b>modulefile</b></code>
also sets functions or aliases for use with the application. A user need only invoke a single
command,
<p>
<center><code><b>module load <application></b></code></center>
<p>
at each login to configure an application/programming environment properly. If you often need an
application environment, place the modules command in your <code><b>.login_user</b></code> and/or
<code><b>.profile_user</b></code> shell startup file.
<p>
Most of the package directories are in <b><code>/usr/local/apps</code></b>
(<b><code>$APPS</code></b>) and are named after the package name
(<b><code><app></code></b>). In each package directory there are subdirectories that contain
the specific version of the package. The <code><b>APPS</b></code> directory structure is shown in
the diagram below:
<p>
<center><img src="/resources/userguides/lonestar/lonestar_apps_dir_structure.gif" alt="Lonestar Applications Directory Structure">
<p>
<font class=footer>TACC Applications Directory Structure</font></center>
<p>
The directory structure for the fftw package is shown below. The directory fftw in
<code><b>/usr/local/apps</b></code> contains 3 different version directories for the package:
2.1.3, 2.1.5 and version 3.0. Since fftw-2.1.5 is the present default version, a fftw link is
created to the default, the fftw-2.1.5 subdirectory.
<p>
<center><img src="/resources/userguides/lonestar/lonestar_fftw.gif" alt="Example FFTW Applications Directory Structure">
<p>
<font class=footer>Example FFTW Applications Directory Structure</font></center>
<p>
The directory paths for the different fftw package versions, can be
constructed easily with the help of the <code><b>$APPS</b></code> variable:
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>$APPS/<app>/<app.version></b></code></td>
<td class=text>{path to specific package version}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>$APPS/<app>/<app></b></code></td>
<td class=text>{link to default version}</td>
</tr>
<tr bgcolor=#e4e4e4>
<td><code><b>$APPS/fftw/fftw</b></code></td>
<td class=text>{example, default version directory for fftw}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>$APPS/fftw/fftw-2.1.3</b></code></td>
<td class=text>{example, directory for earlier version of fftw}</td>
</tr>
</table>
<p>
The fftw package requires several environment variables that point to its home, libraries, include
files, and documentation. These can be set in your environment by loading the fftw module:
<p>
<center><code><b>module load fftw</b></code></center>
<p>
The details of the environmental changes are in the <b><code>modulefile,
/usr/local/opt/modules/modulefiles/fftw</code></b>. To see a list of available modules and a
synopsis of a <b><code>modulefile</code></b>'s operations, execute:
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>module available</b></code></td>
<td class=text>{lists modules}</td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>module help <app></b></code></td>
<td class=text>{lists environment changes performed for <app>}</td>
</tr>
</table>
<p>
During upgrades, new <b><code>modulefiles</code></b> are created to reflect the changes made to
the environment variables. TACC will always announce upgrades and module changes in advance.
<p>
Another feature of modules is the ease in changing the environment for experimenting with new
updates or backing down to older application versions. TACC will often make a link from
<b><code><app>.new</code></b> to the updated package modulefile
(<code><b><app>.<new-version></b></code>) that has not become the default version
yet. Also, the retired default <code><b>modulefile</b></code> is often linked to
<code><b><app>.old</b></code>. This makes it easier for users to change to new or old
environments with the commands:
<p>
<table align=center>
<tr bgcolor=#e4e4e4>
<td><code><b>module swap <app> <app>.old</b></code></td>
</tr>
<tr bgcolor=#f0f0f0>
<td><code><b>module swap <app> <app>.new</b></code></td>
</tr>
</table>
<p>
(If the app module has not been loaded, then it is only necessary to load the new or old version;
e.g. <b><code>module load <app>.old</code></b>.)
<p>
For more information on modules and a description of how to build
<b><code>modulefiles</code></b>, check out the man pages and the following
<a href="http://www.tacc.utexas.edu/services/userguides/guide_config/body.php?sys=pwrE#modules">link</a>.
<p>
For information on customizing your login, check out this
<a href="http://www.tacc.utexas.edu/services/userguides/login/">link</a>.
</div>
Last modified: October 04 2006 09:47:16. <p>
<hr color="#013166" width="100%" size=2>
<p>
<table cellpadding=0 cellspacing=0 align=center height=60>
<tr>
<td align=center valign=middle class=title><a name=filesystems>File Systems</a></td>
</tr>
</table>
<div class="text">
The TACC HPC platforms have several different file systems with distinct storage characteristics.
There are predefined, user-owned directories in these file systems for users to store their data.
Of course, these file systems are shared with other users, so they are managed by either a quota
limit, a purge policy (time-residency) limit, or a migration policy.
<p>
To determine the size of a file system, <b><i>cd</i></b> to the directory of interest and execute
the "<b><code>df</code></b>" command with the syntax:
<p>
<table align=center>
<tr>
<td><code><b>df -k .</b></code></td>
</tr>
</table>
<p>
or simply execute it without the "<code><b>dot</b></code>" to see all file systems. In the example
below the file system name appears on the left, and the used and available space
(<b><code>-k</code></b>, in units of 1KBytes) appear in the middle columns followed by the percent
used:
<p>
<table cellspacing=10 align=center>
<tr>
<td><code><b>% df -k .</b></code></td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td><code><b>File System</b></code></td>
<td><code><b>1k-blocks</b></code></td>
<td><code><b>Used</b></code></td>
<td><code><b>Available</b></code></td>
<td><code><b>Use%</b></code></td>
<td><code><b>Mounted on</b></code></td>
</tr>
<tr>
<td><code><b>/dev/vg/home</b></code></td>
<td><code><b>8256952</b></code></td>
<td><code><b>6675732</b></code></td>
<td><code><b>1161792</b></code></td>
<td><code><b>86%</b></code></td>
<td><code><b>/home</b></code></td>
</tr>
</table>
<p>
To determine the amount of space occupied in a user-owned directory, <b><code>cd</code></b> to the
directory and execute the <b><code>du</code></b> command with the <b><code>-sb</code></b> option
(s=summary, b=units in bytes):
<p>
<table align=center>
<tr>
<td><code><b>du -sb</b></code></td>
</tr>
</table>
<p>
To determine quota limits and usage on <code><b>$HOME</b></code>, execute the quota command
without any options (from any directory):
<p>
<table align=center>
<tr>
<td><code><b>quota</b></code></td>
</tr>
</table>
<p>
The four major file systems and directories available on TACC HPC machines are:
<dl>
<dt><font class="field">home directory</font></dt>
<dd>The system automatically changes to a user's home directory at login
and this is the recommended location to store your source codes and build your executables.
<br>
Lonestar quota limit is <b><code>200MB</code></b>.
<br>
Use <b><code>$HOME</code></b> to reference your home directory in scripts.
<br>
Use <b><code>cd</code></b> to change to <b><code>$HOME</code></b>.</dd>
<p>
<dt><font class="field">work directory</font></dt>
<dd>Store large files and perform most of your job runs from this file system.
This file system is accessible from all the nodes.
<br>
Lonestar quota limit is <b><code>500GB</code></b>.
<br>
Use <b><code>$WORK</code></b> to reference your work directory in scripts.
<br>
Use <b><code>cdw</code></b> to change to <code><b>$WORK</b></code>.
<br>
<b> Files are purged if they have not been accessed within 10 days.</b></dd>
<p>
<dt><font class="field"> scratch or temporary directory</font></dt>
<dd>This is the directory on each node where you can store files and perform local I/O for the
duration of a batch job. Often, in batch jobs it is more efficient to use and store files
directly on <code><b>$WORK</b></code> (to avoid moving files from <code><b>scratch</b></code>
at the end of a job). The scratch directory is only available for the duration of a job.
<br>
Lonestar file system size and limit is <b><code>25GB</code></b> on compute nodes.
<br>
Use <b><code>$SCRATCH</code></b> to reference the scratch directory in scripts.</dd>
<!-- No such command available on lonestar or tejas Use <b><code>cds</code></b> to change to <b><code>$SCRATCH</code></b>.</dd> -->
<p>
<dt><font class="field">san directory</font></dt>
<dd>The SAN directory is available on login nodes (front-ends) of Lonestar, Wrangler, and Champion.
Space on the SAN is an allocatable resource; that is, space is not automatically available
to a project, the Principal Investigator must request space on this file system.
<br>
Lonestar project limit is <b><code>250GB</code></b>.
<!-- Use <b><code>$SAN</code></b> to reference project directory in scripts. -->
<br>
The top level directory of the san is <b><code>/san/hpc</code></b>.</dd>
<p>
<dt><font class="field">archive directory</font></dt>
<dd>Store permanent files here. This file system has "archive" characteristics (see below). The
access speed is low relative to the work directory.
<br>
Lonestar archive has <b>no space limit</b>.</b>.
<br>
Use <b><code>$ARCHIVE</code></b> to reference your archive directory in scripts.
<br>
Use <b><code>cda</code></b> to change to $ARCHIVE.</dd>
</dl>
<p>
<font class="sub">$HOME Directories</font>
<p>
A user's home directory is the place to store files that are routinely used in development and
day-to-day work. If the output files from production runs are small, then it is reasonable to
store them in <code><b>$HOME</b></code>. Home directories are backed up daily; so, if you
accidentally remove a critical file, submit a request using the
<a href="http://portal.tacc.utexas.edu" target="_blank">Portal</a> to recover the last saved version
(include the full path name of the file(s) or directory, as well as the machine name).
Since the home file system is of limited size, a 200 megabyte quota limit is imposed on every user
(the quota limit is machine specific).
<p>
<font class="sub">$WORK Directories</font>
<p>
The <i>work</i> file system is configured with fast disks on TACC machines and should, therefore,
be used when I/O performance significantly affects program performance. Work can also be used to
store large files temporarily. The work file system may be as simple as SCSI disks arranged in a
RAID-3/4/5 configuration and exported through NFS to compute nodes. On Champion
parallel and global access to <i>work</i> is made through GPFS, a General
Parallel File System. On Lonestar <i>work</i> is an
(<a href="http://www.tacc.utexas.edu/services/userguides/lonestar3/work.php" target=_blank>LUSTRE</a>) File System; it can be used for parallel I/O;
and it is accessible from the development/login nodes and all compute nodes.
<p>
The <b>files in work are NOT backed up</b> and are temporary. Files that are corrupted or
accidentally removed are not recoverable. Files that are not accessed within <b>10</b> days are
removed. (Each night a "scrubber" program evaluates access times for every file and removes
outdated files. The scrubber does not remove any files if the user has a running batch job.)
Reminder: for permanent storage use the TACC data archive. To see a daily log of the files that
have been removed from your directory, view the file:<br>
<center><code>/work/reaver/$USER_YYYYMMDD</code></center><br>
where $USER is evaluated as your login name, and YYYY, MM, and DD are the year(4 digits), month(2 digits), and day(2 digits) of the log date.
<p>
<font color="#C32736">PLEASE NOTE: TACC staff may delete files from work if the work file system becomes full and directories consume an inordinately large amount of disk space, even if files are less than 10 days old. A full work file system inhibits use of the file system for ALL users. The use of programs or scripts to actively circumvent the file purge policy will not be tolerated.
</font>
<p>
<font class="sub">$SCRATCH Directories</font>
<p>
<b><code>$SCRATCH</code></b> is a "scratch" directory on each compute node, where a batch job can
store files or perform local I/O. The <b><code>$SCRATCH</code></b> directory on each node is scrubbed
after the job completes, so a job script's final commands should copy any valuable data to
<i>home</i> or <i>work</i>. The scratch directory is the /tmp file system on the Linux clusters.
On the Lonestar front-end, $SCRATCH is /tmp.
If you use $SCRATCH on the front end, please create your own subdirectory, e.g. $SCRATCH/$USER,
to isolate your files within a single directory.
<p>
<font class="sub">$SAN Directory</font>
<p>
The TACC SAN is a Storage Area Network that is accessible from the front-ends of Lonestar, Wrangler, and Champion.
The <code><b>/san/hpc/<i><project_name></i></b></code> directories are for
projects that have been awarded (allocated) long-term space. The present configuration
has ~5TB space for persistent, project-oriented storage. For more details read:
<p>
<center><a href="/services/userguides/san/san.html" target="_blank">More on SAN</a>.</center>
<font class=sub>$ARCHIVE Directories</font>
<p>
For long term file storage, use the archive file system (<code><b>$ARCHIVE</b></code>). This file
system physically resides on an SGI Origin 2000 (archive.tacc.utexas.edu), a machine dedicated to
supporting the archive file system.
<p>
A user's archive directory is available on all TACC HPC computers because the archive file system is mounted as
<code><b>/archive</b></code> on the front-end of each system.
It appears as a normal UNIX file system but is managed by DMF, SGI's
Data Migration Facility. Files that have not been accessed in 10 days are moved offline
(migrated) to tape via two StorageTek 9310 robots. DMF automatically and transparently performs
the archival and retrieval of files from the tape robot system. When an off-line file is accessed,
DMF automatically retrieves the file while the process that is accessing the files waits. Under
normal circumstances it takes less than a minute for the robots to start streaming a file's data
back to the disk and for the user's process to continue.
<p>
<b>Note: On all TACC production machines (Champion,Lonestar)
<code>$ARCHIVE</code> is only mounted on (available from) the front-end and NOT from any of the
compute nodes. As a consequence, users must be careful to migrate files to and from
<code>$ARCHIVE</code> before jobs are submitted or after completion of their run. Access of any
files located in <code>$ARCHIVE</code> through Loadleveler/LSF script options will result in the job
hanging in an <i>idle</i> state.</b>
<!--
<p>
Whether a file is migrated or not, an "<code><b>ls</b></code>" will always show the file. The
command <b><code>dmls -l</code></b>" shows its migration status: "<code><b>OFF</b></code>" means
offline (migrated), "<code><b>REG</b></code>" means regular (not-migrated), and
"<code><b>DUL</b></code>" means dual mode (migration will occur soon). Below is a sample output
from the "<b><code>dmls -l</code></b>" command:
<p>
<table align=center cellspacing=10 cellpadding=0>
<tr>
<td><code><b>%dmls -l</b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
<td><code><b> </b></code></td>
</tr>
<tr>
<td><code><b>-rw-------</b></code></td>
<td><code><b>l</b></code></td>
<td><code><b>tnek</b></code></td>
<td><code><b>hpc</b></code></td>
<td align=right><code><b>958</b></code></td>
<td><code><b>Apr</b></code></td>
<td align=right><code><b>27</b></code></td>
<td align=right><code><b>1998</b></code></td>
<td><code><b>(REG)</b></code></td>
<td><code><b>data</b></code></td>
</tr>
<tr>
<td><code><b>-rw-------</b></code></td>
<td><code><b>l</b></code></td>
<td><code><b>tnek</b></code></td>
<td><code><b>hpc</b></code></td>
<td align=right><code><b>749731840</b></code></td>
<td><code><b>Nov</b></code></td>
<td align=right><code><b>10</b></code></td>
<td align=right><code><b>11:10</b></code></td>
<td><code><b>(DUL)</b></code></td>
<td><code><b>y.tar</b></code></td>
</tr>
<tr>
<td><code><b>-rw-------</b></code></td>
<td><code><b>l</b></code></td>
<td><code><b>tnek</b></code></td>
<td><code><b>hpc</b></code></td>
<td align=right><code><b>46792704</b></code></td>
<td><code><b>Oct</b></code></td>
<td align=right><code><b>5</b></code></td>
<td align=right><code><b>1998</b></code></td>
<td><code><b>(OFF)</b></code></td>
<td><code><b>x.tar</b></code></td>
</tr>
</table>
-->
<p>
When moving files from archive to a local file system (home or work) use <b><code>ftp</code></b>
or <b><code>rcp</code></b> if the files are large (>100MB). The <b><code>cp</code></b> command
transfer rates from an NSF mounted file system are about 5 times slower than a transfer with
<code><b>ftp</b></code> or <code><b>rcp</b></code>.
<p>
<table align=center>
<tr>
<td><code><b>rcp ${ARCHIVER}:$ARCHIVE/myfile $WORK</b></code></td>
</tr>
</table>
<p>
For the fastest large-file transfer to a local work directory from archive, use the command above.
Don't forget to include the archive machine name (<code><b>$ARCHIVER</b></code> is defined for
you), else the <b><code>rcp</code></b> utility will simply use "<code><b>cp</b></code>" to do the
transfer. In some shells, curly braces may be required around the environment variables, e.g.
<code><b>${ARCHIVER}</b></code> when followed by a colon (:).
<p>
<!-- Not needed anymore TM 05/05/04
<font class=sub><code>$TMPDIR</code> Directories</font>
<p>
Each batch job is given a directory for the duration of its run. The file system path to this
directory is set in the <code><b>TMPDIR</b></code> environment variable immediately before the
batch job begins execution. Use your <code><b>$TMPDIR</b></code> directory to hold scratch files
that require a modest amount of I/O throughput or are "write once" files. Note: all files in
<code><b>$TMPDIR</b></code> are scrubbed when a job ends, so it is necessary to move any needed
files in <code><b>$TMPDIR</b></code> to an appropriate directory before the job ends.
<p>
<b>Note: The Clusters, Power4, and PowerEdge systems currently do not have
<code><b>$TMPDIR</b></code> available for this purpose. It's mostly applicable to Cray systems, so
in current production hardware the Cray SV1 is the only machine for which this discussion on
<code><b>$TMPDIR</b></code> is applicable.</b>
<p>
In batch jobs it is a good practice to make <b><code>work</code></b>, <code><b>tmpdir</b></code>,
or <code><b>archive</b></code> your current directory because of the limited space in your
<code><b>home</b></code> directory. The commands "<b><code>cd $WORK</code></b>" and
"<b><code>cd $TMPDIR</code></b>", and "<b><code>cd $ARCHIVE</code></b>" will make your
<code><b>work</b></code>, <code><b>temp</b></code>, or <code><b>archive</b></code> directory the
current working directory. For your convenience in interactive sessions, TACC has defined the
<b><code>cdw</code></b> and <b><code>cda</code></b> "commands" for changing the current directory
to your <code><b>work</b></code> and <b><code>archive</b></code> directory. In all batch jobs you
should specify the maximum amount of disk space a run will use. As an example for NQS (Cray) jobs,
users should specify with the <b><code>-lF XGB</code></b> option, where X is an integer number.
__>
<!--
For NQS (Cray), PBS (IA-32/64), and LoadLeveler (IBM) the options are <b><code>-lF XGB</code></b>,
<b><code>-lfile=Xgb</code></b>, and <b><code>-???</code></b>, respectively; where X is an integer
number. Execute <code>"qconfig"</code> on any system to see the file limits for the queues.
-->
<p>
<!-- this isnt working anyway, so its getting cut -evan 2/23/06
Below is a diagram that summarizes the file lifetimes and space limits of the user file systems.
<p>
<center><img src="/resources/userguides/lonestar/lonestar_filesys.png" alt="Lonestar File System"></center>
<p>
-->
<!-- Not needed now 2-21-06 Evan
<font class="sub">HPSS access for NPACI Users</font>
<p>
NPACI users can access the archival storage at SDSC, the High Performance Storage Systems (HPSS).
For more information and details on HPSS please see the
<a href="http://www.npaci.edu/HPSS/hpss.html" target=_blank>NPACI HPSS User Guide</a>.
<p>
The supported interface to HPSS at TACC is <b><code>hsi</code></b>. <b><code>hsi</code></b>
supports wildcards for local and HPSS pathname pattern matching and provides recursion for many
commands, including the ability to store, retrieve, and list entire directory trees or change
permissions on entire trees. It may be used interactively or in batch mode and may be included in
UNIX pipes. <b><code>hsi</code></b> is also especially useful for NPACI users with accounts on
multiple platforms since it is the usual interface to the SDSC HPSS system from most NPACI
machines.
<p>
In order <a href="http://www.npaci.edu/HPSS/HSI/hsi.html" target=_blank>to use
<code><b>hsi</b></code></a>, users must have a Distributed Computing Environment (DCE) keytab. It
must be named "<code><b>.ktb_<i><login_name></i></b></code>" and be located in the
<code><b>.private</b></code> directory in <b><code>$HOME</b></code>. Also, the
<b><code>.hsirc</b></code> file should be present, and contain the following lines:
<p>
<pre>
authmethod = keytab
principal = <b><login_name_here></b>
keytab = %H/.private/.ktb_<b><login_name_here></b>
</pre>
<p>
Keytab retrieval and storage is automatically performed on any SDSC machine with the command
"<b><code>usr/local/apps/HPSS/bin/get_hpss_keytab.ksh</code></b>". Once the files are in place
they can be ssh'd to the Power4 machine into appropriate directories (<code><b>$HOME</b></code>
for <code><b>.hsirc</b></code> and <code><b>$HOME/.private</b></code> for the keytab). If you need
a keytab and no longer have access to an SDSC machine, please submit a request through the
<a href="http://www.npaci.edu/Consult/" target=_blank>NPACI Consulting Form</a>. When the request
is approved a keytab file and a <code><b>.hsirc</b></code> file will be installed in appropriate
directories for you on the Power4.
<p>
In batch job scripts you can use "single-line" commands to get/put files from/into the HPSS,
similar to pftp. The syntax for single-line execution is:
<p>
<center><code><b>hsi</b> <i>command1</i><b>[</b><i>command2</i>;<i>command3</i>;<i>...</i><b>]</b></code></center>
<p>
-->
</div>
Last modified: July 19 2007 16:02:27. <p>
<hr color="#013166" width="100%" size=2>
<p>
<!--Top of page Anchor-->
<a name="models"></a>
<table border=0 cellpadding=0 cellspacing=0 align=center height=60px>
<tr>
<td align=center avalign=middle>
<div class="title">Programming Models</div>
</td>
</tr>
</table>
<div class="text">
For both the Intel clusters and the IBM Power5 SMP clusters, there are
a couple of programming models for consideration.
<p>
First, in the distributed memory, message-passing model one may use either
the SPMD or the MPMD programming paradigms. With the former, each processor
loads the same program image and executes but on different data sets. This is
dependent on the local portion of distributed arrays according to their distribution,
array sizes, and number of processors determined at runtime. With the MPMD
paradigm, each processor loads up and executes a different program image and on
different data sets, with a similar set of parameters as the SPMD paradigm.
This distributed memory message passing programming model strictly uses MPI for
its communication. This is advocated as the best model for optimal use
of the clusters and is also suitable for the Power5 nodes.
<p>
The other programming model that is always recommended on the Symmetric Multi-
Processor based nodes like the IBM Power5 nodes or Parallel Vector Processing (PVP) systems,
is the SMP (Shared Memory Programming)
model using either proprietary directives (or pragmas), vendor implementations of
standards like OpenMP, or explicit threaded implementations such as Pthreads.
Here, the major point of distinction is that all of the processors in the
same node share data, hence strict processor to processor message-passing
is unnecessary. Further, additional parallelism can be extracted by use
of threads or multi-threaded implementations of the compilers, libraries,
or tools. With the present state-of-the-art operating systems, compilers,
and tools it is
highly recommended to use OpenMP on SMP based nodes with MPI across
nodes in a hybrid programming environment. There isn't sufficient evidence
at this time to suggest that using OpenMP with MPI in a hybrid
environment on the clusters would be of great benefit from the performance
point of view. However, we will inform users if this changes in the future.
<p>
For further information on OpenMP, MPI and on programming models/paradigms,
please see the <a href="#manuals" class="link3">manuals</a> and
<a href="#packages" class="link3">packages</a> sections for documentation.
<p>
</div>
Last modified: October 04 2006 09:47:16. <p>
<hr color="#013166" width="100%" size=2>
<p>
<!--Top of page Anchor-->
<a name="compilation"></a>
<table border=0 cellpadding=0 cellspacing=0 align=center height=60px>
<tr>
<td align=center avalign=middle>
<div class="title">Compilation</div>
</td>
</tr>
</table>
<!--Table of Contents-->
<table border=0 align=center height=40px>
<tr>
<td nowrap><a href="#serial" class="link2">Serial</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#openmp" class="link2">OpenMP</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#mpi" class="link2">MPI</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#optimization" class="link2">Optimization </a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#libs" class="link2">Loading Libraries</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#limitmemory" class="link2">Limiting Memory Usage</a></td>
</tr>
</table>
<div class="text">
<p>
<a name="serial"><font class="section">Compiling and Running Serial Programs</font></a>
<p>
The compiler commands can be used for both compiling and loading (making an
executable from a <i>".o"</i> object file). The table(s) below lists the syntax for
serial and parallel program compilation.
</div>
<p>
<blockquote>
<font class="sub">Compiling Serial Programs</font>
<p>
<table border=1 cellpadding=2 cellspacing=0 bordercolor="#013166">
<tr>
<td class="field">Compiler</td>
<td class="field">Program</td>
<td class="field">TypeSuffix</td>
<td class="field">Example</td>
</tr>
<tr>
<td class="text">xlc_r</td>
<td class="text">c</td>
<td class="text">.c</td>
<td><tt><b>xlc -c [options] ibmc.c</b></tt></td>
</tr>
<tr>
<td class="text">xlC_r</td>
<td class="text">C++</td>
<td class="text">.cpp, .C</td>
<td><tt><b>xlC -c [options] ibmcpp.C</b></tt></td>
</tr>
<tr>
<td class="text">xlf_r</td>
<td class="text">F77</td>
<td class="text">.f</td>
<td><tt><b>xlf -c [options] ibmfor.f</b></tt></td>
</tr>
<tr>
<td class="text">xlf90_r</td>
<td class="text">F90</td>
<td class="text">.f</td>
<td><tt><b>xlf90 -c [options] ibmf90.f</b></tt></td>
</tr>
</table>
</blockquote>
<div class="text">
<p>
Appropriate program-name suffixes are required for each compiler. By
default an <b>a.out</b> executable is created by the compiler
invocation. The executable may be renamed with the <b>-o</b> option,
and execution is performed by specifying the executable on the command
line. Also <i>"options"</i> denote additional compiler non-default
options and may be added during the compilation.
<p>
<b> Note: For those users migrating from <i> Longhorn </i> Power4
system, for the sake of uniformity, we are now recommending two major
changes. First, use of the re-entrant version of the compiler (or use
of compilers with trailing"_r" symantic notation) for <i> all </i> types of codes:
serial, threaded or parallel. The second change is for building 64-bit
codes for all application. All the threaded and parallel codes see
a clear performance benefit from these changes. Additional details on this
subject is provided below.</b>
<blockquote>
<table border=0 bordercolor="#013166" cellpadding=5 cellspacing=0>
<tr>
<td class="text">Compile/Link code</td>
<td><tt><b>prog.cpp</b> or <b>prog.f</b>, <b>naming the executable prog</b></tt></td>
</tr>
<tr>
<td class="text">C</td>
<td><tt><b>xlc_r -o prog -q64 [other options] prog.c [linker options]</b></tt></td>
</tr>
<tr>
<td class="text">Fortran</td>
<td><tt><b>xlf90_r -o prog -q64 [other options] prog.f [linker options]</b></td>
</tr>
</table>
</blockquote>
<p>
Here <i>"linker options"</i> denote zero or more linker loader options such
as library directories and their names. Also the <i>"option"</i> compiler
options take precedence over the <i>"link loader options</i>."
<p>
To run the above interactive program, execute:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>./prog</b></tt></td>
</tr>
</table>
</blockquote>
<p>
The relative path expression <i>"./"</i> tells the shell to look in the present
working directory for the executable. It is often used to make sure that an
executable of the same name in another directory (as determined by the PATH
environment variable) is not executed. Also, if the <i>"."</i> is not in the PATH
variable it is necessary to use <i>"./"</i> for the shell to find the executable. If
it is in the PATH variable however, just typing the executable name is
suffient to run your program.
<p>
A brief discussion of compiler options, their syntax, and a terse explanation
is given below in the optimization section. In addition, typing in the compiler names
<b> xlc</b> or <b>xlf</b>
etc., will give additional information on all of the compiler options.
Online documentation and user guides for compilers are also available.
<p>
<hr color="#013166" width="50%">
<p>
<a name="openmp"><font class="section">Compiling OpenMP Programs</font></a>
<p>
All the nodes are eight-way symmetric multiprocessors using shared memory, and as
such are a perfect model for using Shared Memory Programming (SMP) paradigm. All the nodes
have an equal amount of shared memory, roughly 14 GBytes, available for application use.
Using any of the nodes as an SMP, they may be effectively used in
combination with thread-safe compilers with appropriate compiler options and
setting run-time variables, and libraries optimal for SMP environment.
<p>
The compilers are used similar to serial compilation, as given below, except in the slight change
in the naming syntax. As an example, a program compiled using the SMP environment by the
<i>xlf90_r</i> compiler is:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>xlf90_r -qsmp=noauto:omp -qnosave -o prog [options] progOMP.f [linker options]</b></tt></td>
</tr>
</table>
</blockquote>
<P>
Use "<i>-qsmp=noauto:omp</i>" when compiling programs with OpenMP directives
or pragmas, to disable automatic parallelization by the compiler. The
"<i>-qnosave</i>" option sets all local variables as automatic, and is
required to ensure correct behavior of Fortran programs that
call routines within a parallel region. Also note that <i>-qhot</i> is
turned on, by "<i>-qsmp=auto</i>" but not by "<i>-qsmp=omp</i>, so make
sure to include the correct SMP flag for high order transformation.
The option <i>-qnohot</i> should be used for consistency with
directive/pragma based SMP runs.
Preferably any optimizations including <i> -qhot </i> should be tested
in a single threaded manner
before using <i>-qsmp</i>, whenever possible.
<p>
Note that unlike the serial programs,
<b> any programs using threads, or parallel MPI calls, must use the reentrant
version of the compiler or the compiler using the _r symantic notation </b>.
Here the <b> xlf90_r </b> is the <b> xlf90 </b> compiler which links to the
thread-safe components, as well as other threaded version of the libraries.
<p>
Similarly, any compiler can be used in its place with the same syntax change. The
compiler options are the same as the serial ones. The compiler options specifically
using the SMP environment either automatically and/or with/without directives
must be included as part of the <i> linker options </i>. A list of such
options with description are given below <a href="#optimization" class="link3">
Compiler Options</a>. The use of SMP optimized libraries are given below
<a href="#libraries" class="link3"> Performance Libraries </a>.
<p>
<hr color="#013166" width="50%">
<p>
<a name="mpi"><font class="section">Compiling Parallel Programs with MPI</font></a>
<p>
The compiler commands can be used for both compiling and loading (making an
executable from a <i>".o"</i> object file). IBM MPI shell scripts for Power5
servers are prefixed with <b>mp</b>. The table(s) below lists the syntax for
serial and parallel program compilation.
<p>
<blockquote>
<font class="sub">Compiling Parallel Programs</font>
<p>
<table border=1 cellpadding=2 cellspacing=0 bordercolor="#013166">
<tr>
<td class="field">Compiler</td>
<td class="field">Program</td>
<td class="field">TypeSuffix</td>
<td class="field">Example</td>
</tr>
<tr>
<td class="text">mpcc</td>
<td class="text">c</td>
<td class="text">.c</td>
<td><tt><b>mpcc_r [options] ibmc.c</b></tt></td>
</tr>
<tr>
<td class="text">mpCC_r</td>
<td class="text">C++</td>
<td class="text">.cpp, .C</td>
<td><tt><b>mpCC_r -cpp -q64 [other options] ibmcpp.C</b></tt></td>
</tr>
<tr>
<td class="text">mpxlf_r</td>
<td class="text">F77</td>
<td class="text">.f</td>
<td><tt><b>mpxlf_r -q64 [other options] ibmfor.f</b></tt></td>
</tr>
<tr>
<td class="text">mpxlf90_r</td>
<td class="text">F90</td>
<td class="text">.f</td>
<td><tt><b>mpxlf90_r -q64 [other options] ibmf90.f</b></tt></td>
</tr>
</table>
</blockquote>
<p>
Appropriate program-name suffixes are required for each compiler. By
default an <b>a.out</b> executable is created by the compiler invocation. The
executable may be renamed with the <b>-o</b> option, and execution is performed by
specifying the executable on the command line. Also, <i>"options"</i> denote additional
compiler non-default options and may be added during the compilation. For C++
programs in MPI, the <b>mpCC_r</b> needs
to be used with <b>-cpp</b> option, along with any other options of the
user's choice.
<p>
<blockquote>
<table border=0 bordercolor="#013166" cellpadding=5 cellspacing=0>
<tr>
<td class="text">Compile/Link code</td>
<td><tt><b>prog.c</b> or <b>prog.f</b>, <b>naming the executable prog</b></tt></td>
</tr>
<tr>
<td class="text">C</td>
<td><tt><b>mpcc_r -o prog -q64 [other options] prog.c [linker options]</b></tt></td>
</tr>
<tr>
<td class="text">Fortran</td>
<td><tt><b>mpxlf90_r -o prog -q64 [other options] prog.f [linker options]</b></tt></td>
</tr>
</table>
</blockquote>
<p>
Here <i>"linker options"</i> represents zero or more linker loader options.
Also the <i>"option"</i> compiler options take precedence over the <i>"link loader
options</i>."
<p>
As we briefly stated above,
from performance standpoint it is recommended that all MPI programs
use the 64-bit version of the MPI libraries. These 64-bit MPI libraries are
thread-safe, hence the re-entrant version of the compilers should be used in
conjunction. For those needing to use the 32-bit MPI libraries do not need to
use the <b> _r </b> semantic in the compiler invocation but will also see
poorer performance from the 32-bit versions of these libraries. There are
additional details on this issue later in the section for
<a href="#mpi_parallel" class="link3"> MPI Parallel Programming Environment </a>.
<p>
To run the above compiled program interactively, execute:
<p>
<blockquote>
<table border=0>
<tr>
<td nowrap><tt><b>% poe ./a.out -procs n</b></tt></td>
</tr>
</table>
</blockquote>
<p>
A brief list of compiler options, their syntax, and a terse
explanation is given below in the optimization section. In addition,
<b>man mpcc</b> or <b>man mpxlf</b> etc. provide additional details on all of the
compiler options. User guides and online documentation provide
additional resources.
<p>
<hr color="#013166" width="50%">
<p>
<a name="optimization"><font class="section">Program Optimization for Serial and Parallel
Programming using OpenMP and MPI</font></a>
<p>
The MPI shell scripts can compile programs using the normal serial compiler options. Thus,
as an example, any of the compiler flags normally accepted by the <b>xlc</b> command can
also be used on <b>mpcc</b>. Given below are some of the common serial compiler options
with descriptions.
</div>
<p>
<table border=1 cellpadding=2 cellspacing=0 bordercolor="#013166">
<tr>
<td class="field">Compiler Options</td>
<td class="field">Description</td>
</tr>
<tr>
<td class="text">-O3</td>
<td class="text">performs some compile time and memory intensive optimizations in
addition to those executed with -O2. They also have the potential to alter semantics
of a user's program.
</td>
</tr>
<tr>
<td class="text">-qarch=pwr5 (or auto)</td>
<td class="text">specifies instruction-set architecture for Power5 hardware</td>
</tr>
<tr>
<td class="text">-qtune=pwr5 (or auto)</td>
<td class="text">produces object files optimized for Power5 architecture, including
instruction scheduling and memory hierarchy.</td>
</tr>
<tr>
<td class="text">-qhot</td>
<td class="text">high order transformations to maximize efficiency of loops and array
language.</td>
</tr>
<tr>
<td class="text">-qcache=auto</td>
<td class="text">specifies cache configuration for compiling machine or relevant
executing environment.</td>
</tr>
<tr>
<td class="text">-qipa</td>
<td class="text">optimizes by performing analysis across procedures.</td>
</tr>
<tr>
<td class="text">-bmaxdata:<bytes></td>
<td class="text">Specifies the maximum amount of space to reserve for the program data
segment for programs where the size of these regions is a constraint (units=bytes).</td>
</tr>
</table>
</blockquote>
<div class="text">
<p>
<b> Warning: </b> It has come to our attention that certain codes are
giving erroneous results on use of the <b> -qhot </b> (high-order
transformation) compiler option. Note that, for optimization levels
above -O3, -qhot is used as default. Please verify your results for
correctness when compiling at that level or when using -qhot
directly. Using the option <b> -qnohot </b> avoids using any high-order
transformation even at -O4 or -O5 compiler option level. This bug has been
brought to the IBM compiler groups attention. We will apply any future patches
for this issue.
<p>
In addition, in a Shared Memory Programming (SMP) environment, some of the additional
compiler options, which must be used along with <i>runtime</i> environment variables and
SMP optimized libraries, are listed below.
</div>
<p>
<table border=1 cellpadding=2 cellspacing=0 bordercolor="#013166">
<tr>
<td class="field">SMP Compiler Options</td>
<td class="field">Description</td>
</tr>
<tr>
<td class="text">-qsmp</td>
<td class="text">Enables use of shared memory parallelization by <i>user
directives</i>, OpenMP or explicitly or by MPI calls.</td>
</tr>
<tr>
<td class="text">-qsmp=omp</td>
<td class="text"><i> Explicit</i> or <i>User directed</i> OpenMP environment or IBM
proprietory directives (or pragmas for C/C++).</td>
</tr>
<tr>
<td class="text">-qsmp=auto</td>
<td class="text"><i>Automatic</i> parallelization enabled.</td>
</tr>
<tr>
<td class="text">-qsmp=explicit</td>
<td class="text"><i>Explicit</i> parallelization by use of <i>Pthreads</i></td>
</tr>
</table>
</blockquote>
<div class="text">
<p>
For further details on additional compiler options on both IBM serial compilers and their
respective MPI scripts, please see the
<a href="/resources/user_guides/#hpc" target="_top">online documentation</a> and <i>man</i> pages.
<!--
<p>
The <b>"Engineering and Scientific Subroutine Library"</b> (ESSL) and its parallel variant PESSL
are the optimized libraries for this architecture. Both of these libraries can be used for
parallel programs. These libraries have functionality which include amongst others, Basic Linear
Algebra Subprogram (BLAS) for basic matrix/vector operations, Scalable Linear Algebra PACKage for
linear algebraic equations solution and eigensystems analysis, and Fourier Transforms. These
libraries can be linked by the loader with the <i>"-lesslsmp"</i> and <i>"-lpesslsmp"</i>
options.
<p>
In addition, another library that is different in functionality from the (P)ESSL family of
libraries and recommended for use is the <b>"Mathematical Acceleration Subsystem"</b> (MASS)
library. It consists of libraries of tuned mathematical intrinsic functions that are available in
two flavors, <i>serial</i> and <i>vectorised</i>. The scalar library contains an accelerated set
of frequently used math intrinsic functions in the AIX system library <b>libm.a</b>, and can be
used by linking in <i>"-lmass"</i> option with the loader. The MASS vector library contains
vector functions that will run on the entire IBM pSeries and RS/6000 families. It can be used by
linking in <i>"-lmassvp4"</i> option. The following is an example of how to compile and link a
parallel program with calls to a vector MASS library function.
<p>
</div>
<blockquote>
<table>
<tr>
<td nowrap><tt><b>mpxlf90 -o test_mass -O3 -qarch=pwr5 -qtune=pwr5
test_mass.f -lmassvp5</b></tt></td>
</tr>
</table>
</blockquote>
<div class="text">
<p>
Please see online documentation at
<a target="_top" href="http://www-1.ibm.com/servers/eserver/pseries/softare/sp/essl.html"
class="link3">www-1.ibm.com/servers/eserver/pseries/software/sp/essl.html</a>
and <a target="_top" href="http://www.rs6000.ibm.com/resource/technology/MASS/"
class="link3">www.rs6000.ibm.com/resource/technology/MASS/</a>
for further reference and information on these libraries.
-->
<p>
<hr color="#013166" width="50%">
<p>
<a name="limitmemory"><font class="section">Limiting Memory Usage</font></a>
<p>
On the <i> Champion </i> Power5 system, the default object mode is 64-bit, which means
that in the default environment you will build executables that run in 64-bit address mode.
Large-memory applications use the -q64 compiler/loader option to enable access to
essentially all of the available memory of any of the TACC Power5 nodes. All of the
third-party applications will be built 64-bit, by default. The 32-bit versions of these
applications will only be built for users who need them for dependencies in there legacy code.
<p>
The job memory-per-task limit specified in the ConsumableMemory
resource statement is used by LoadLeveler to assign an appropriate
queue to a job proportional to the number of parallel tasks and global
memory required by the job. It does not actually limit the memory used
by the executable on the Champion system.
<p>
Hence, large-memory applications (that use the -q64 option), in
particular those that use dynamic memory, should be compiled not to
exceed their "fair share" of physical memory per task (processor).
<pre>
<b> Compile with -bmaxdata=1887000000 option </b>
</pre>
Exceeding this limit will cause the executable to swap, and in some
cases the node will crash when the total virtual memory requested
exceeds the physical memory plus the available swap space (the swap
disk space has been configured to equal the physical memory size). To
avoid this problem, we ask developers to use the maxdata loader option
to limit the memory used by executables. Note that for the new Power5
<i> Champion </i> system this limit will be enforced where the node usage is
being shared. For users in dedicated mode, this memory limit will not
be strictly enforced and monitored, but users should be aware of the performance impact.
<p>
<!-- For MPI executables the ConsumableMemory and corresponding maxdata values for each task are listed below:
<blockquote>
<table border=1 cellpadding=2 cellspacing=0 bordercolor="#013166">
<tr>
<th colspan=3>MPI Applications -- Task Memory-Limit Enforcement</th>
</tr>
<tr>
<th>ConsumableMemory</th><th>maxdata limit</th><th>System</th>
</tr>
<tr>
<td>1800MB</td> <td> -bmaxdata=1887000000</td> <td>p655, HPC P690 (lh4, lh16); normal/high queue</td>
</tr>
<tr>
<td>1024MB</td> <td> -bmaxdata=1073000000</td> <td>lonestar; interactive & serial/development queue</td>
</tr>
<tr>
<td>3700MB</td> <td> -bmaxdata=3879000000</td> <td>Turbo p690 (lh32); normal/high queue<br>
Make sure you set ConsumableMemory to 3700MB</td>
</tr>
</table>
</blockquote>
<p>
Note, 1800MB is the fair-share memory per task for the 4-way P655 nodes (totals to 7,200MB for 4 tasks ) and the
HPC P690 nodes (totals to 28,800MB for running 16 tasks); 1024MB is
and appropriate value for interactive an the serial/development queue, that is, for executables running on
the lonestar (login) node; and 3700MB is the fair-share memory for the Turbo P690 (totals to 118400MB for running 32 tasks)
Applications that use OMP should use a ComsumableMemory value equal to the total memory
that all threads in the application will consume.
<p>
The maxdata option doesn't need to be included when files are
compiled (e.g. xlf90 -q64 -c test.f), but it is absolutely necessary for it to be
included on the loader statement, that is, the statement that makes the executable
(e.g. xlf90 -q64 -bmaxdata=1887000000 -o a.out test.o). -->
<p>
<hr color="#013166" width="50%">
<p>
<a name="libs"><font class="section">Loading Libraries</font></a>
<p>
Some of the more useful load flags/options are listed below. For a more comprehensive list, check
out the <i>ld</i> manpage.
<ul>
<li>Use the <b><code>-l </b></code> loader option to link in a library at load time: e.g.
<p>
<dd><code><b>xlf90 prog.f -l<i>name</i></b></code></dd>
<p>
This links in the library <i>libname.a</i> provided it is found in <i>ld</i>'s library search
path.</li>
<p>
<li> To add a directory to the library search path, use the <b><code>-L</code></b> option, e.g.
<p>
<dd><code><b>xlf90 prog.f -L/usr/local/mass -lmass</b></code></dd>
<p>
In the above example, the MASS library linked in by the user is not in the default search
path, so the "<i>-L</i>" option must be specified to point to the <i>libmass.a</i>
directory.</li>
<p>
<li>To change the name of an external symbol, use the <b><code>-brename</b></code> flag at
loadtime. For example, the command:
<p>
<dd><code><b>xlc prog.c -brename:.fortran_func,.FORTRAN_FUNC</b></code></dd>
<p>
will rename the symbol fortran_func to its upper-case equivalent.
<p>
By default, all Fortran symbol names are converted to lowercase. In contrast, C and C++ are
case case-sensitive so upper-case function names remain upper-case. The above loader directive
is useful in insuring compatibility between Fortran and C. In the previous example, references
to <i>FORTRAN_FUNC</i> in a C code which calls a Fortran function will be renamed to
<i>fortran_func</i>.</li>
<p>
<li>To produce a loadmap, use <b><code>-bsxref:filename</code></b> as shown in the example below:
<p>
<dd><code><b>xlf90 prog.f -bsxref:map -lsomelib</b></code></dd>
<p>
This command generates address maps for the object file prog.o and places the output in
alphabetical order to the file "map" in the current working directory.</li>
</ul>
Last modified: October 04 2006 09:47:16. <p>
<hr color="#013166" width="100%" size=2>
<p>
<!--Top of page Anchor-->
<a name="running"></a>
<table border=0 cellpadding=0 cellspacing=0 align=center height=60px>
<tr>
<td align=center avalign=middle>
<div class="title">Running Code</div>
</td>
</tr>
</table>
<!--Table of Contents-->
<table border=0 align=center height=40px>
<tr>
<td nowrap><a href="#runtime" class="link2">
Runtime Environment</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#interactive" class="link2">
Running Interactive Programs</a></td>
<td><img src="/images/star.gif"></td>
<td nowrap><a href="#batch" class="link2">
Running Batch Code</a></td>
</tr>
</table>
<div class="text">
<p>
<a name="runtime"><font class="section">
Runtime Environment</font></a>
<p>
<!--Table of Contents-->
<img src="/images/star.gif"><a href="#parallel" class="link2">
Parallel Environment</a>
<br>
<img src="/images/star.gif"><a href="#mpi_parallel" class="link2">
MPI</a>
<br>
<img src="/images/star.gif"><a href="#shared" class="link2">
OpenMP</a>
<!--<br>
<img src="/images/star.gif"><a href="#authorization" class="link2">
Authorization</a> -->
<p>
<a name="parallel"><font class="sub">Parallel Environment</font></a>
<p>
The Parallel Environment (PE) from IBM is designed for the development and
execution of parallel C, C++, and Fortran programs and has the following:
<p>
<ul>
<li>Parallel Operating Environment for submitting and managing jobs;</li>
<li>Message passing libraries (MPI and MPL);</li>
<li>Parallel debugger <b><i>pdbx</i></b>; and</li>
<li>Xprofiler for analyzing a parallel application's performance.</li>
</ul>
<p>
<font class="field">Parallel Operating Environment</font>
<p>
The Parallel Operating Environment (POE) is one part of the Parallel
Environment. It can be looked upon as a "user interface" to PE and has an
interactive parallel shell with a syntax similar to ksh.
<p>
<!--
<font class="field">POE Definitions</font>
<p>
<dl>
<dt>Node</dt>
<dd>An independent computing element. Each node has its own memory, local
disk space, and operating system and can have one or more processors.</dd>
<p>
<dt>Pool</dt>
<dd>An arbitrary collection of nodes assigned by the system managers. Pools
are used to separate nodes into different groups with different functions. One
node may be designated as a "login" node to do submit jobs or to run interactive
jobs, while other nodes may be reserved for batch jobs.</dd>
<p>
<dt>Home Node</dt>
<dd>The node designated for login, running interactive jobs, or submitting batch
jobs.</dd>
<p>
<dt> Manager</dt>
<dd>A batch scheduling software. This functionality is provided by LoadLeveler.</dd>
</dl>
<p> -->
<font class="field">Setting the Environment Variables</font>
<p>
There are two ways to configure the way a parallel program is executed --
with environment variables or command-line arguments. The POE environment
variables are set using either <b><i>setenv</i></b> (csh/tcsh) or
<b><i>export</i></b> (ksh, bsh) commands and can be set:
<p>
<ul>
<li>at the command prompt;</li>
<li>in the "dot" files (.cshrc_user, .profile_user); or</li>
<li>in the job script file.</li>
</ul>
<p>
The environment variables can be over-ridden with POE command line flags. The
environment variable names are the same as the command line option names, but
they start with <i>"MP_"</i> and are all in upper case. For example, the
command line flag -procs has the corresponding environment variable
<i>"MP_PROCS."</i>
The following are a few, typical POE command line flags:
<p>
<dl>
<dt>Set number of nodes or tasks</dt>
<p>
<dd><ul>
<li>procs - number of task processes</li>
<p>
<li>nodes - number of physical nodes on which to run parallel tasks</li>
<p>
<li>tasks_per_node - number of tasks to be run on each of the
physical nodes</li>
</ul> </dd>
</dl>
<p>
Those interested in using POE command line flags exclusively,
instead of using them through job scripts, should learn about the remaining flags
<A target="_top" HREF="http://publibfp.boulder.ibm.com/epubs/pdf/a2274251.pdf"
class="link3">viewed here</A>.
Those interested in using the environment variables through the use of Loadleveler
job scripts, should go <a href="#batch"> here.</a>
<!--
</ul></dd>
<p>
<dt>Nodes allocation (by user or LoadLeveler)</dt>
<p>
<dd><ul>
<li>MP_RESD - specifies whether or not LoadLeveler should be used to allocate
nodes -- valid values are <i>"yes"</i> or <i>"no"</i></li>
<p>
<li>MP_RMPOOL - specifies the pool number for non-specific node allocation using
LoadLeveler</li>
<p>
<li>MP_HOSTFILE - specify the file with the list of nodes that one wants
allocated for a job</li>
</ul></dd>
<p>
<dt>Communications</dt>
<p>
<dd><ul>
<li>MP_EUILIB - specifies which of two protocols should be used
for task communications -- valid values are <i>"us"</i> for
User Protocol and <i>"ip"</i> for Internet Protocol</li>
<p>
<li>MP_EUIDEVICE - specifies which network adapter should be used for
communications since a node may be physically connected to different
networks -- valid values are <i>"en0"</i> (ethernet), <i>"fi0"</i> (FDDI), <i>"tr0"</i>
(token-ring), <i>"css0"</i> (SP switch), or <i>"csss"</i> (SP switch double
adapter)</li>
</ul></dd>
</dl>
<p>
-->
<center><hr color="#013166" width="50%"></center>
<p>
<a name="mpi_parallel"><font class="sub">MPI Parallel Programming Environment</font></a>
<p>
The IBM Power5 nodes are connected to each other via the high
performance <b> singly-linked Federation or High Performance Switch (HPS) interconnect.</b> All
off node communication is available for use through the IP protocol
as well as the recommended <b><i> User Space (US) protocol </b></i> for better performance
over the HPS switch using both switch interfaces or adaptors on each
node <b> (device is "sn_all")</b>. Users can thus run their MPI or OpenMP
applications with the HPS switch interconnect with either protocol
submitted through their batch scripts. In addition, the MPI implementation
now uses threads instead of signals for asynchronization activities.
<!--(Those who need to use all
four nodes for dedicated benchmarking or other special cases should
first make contact with TACC personnal -- this option is not
generally available with the conventional job scripts.) <p>
-->
<p>
MPI can use several protocols for communication between tasks. IP
(internet protocol) can be used for communication between tasks on a
node or on different nodes. Aside from relatively high latencies, this
also incurs high overhead from use of the slow IP protocol. This
availability is a useful alternative on such systems which do not
have the SP interconnect switch, since it is the only mode of
communication protocol for ethernet, GigE networks etc., for MPI
tasks. Off node communication between tasks is fastest when the US
(user space) protocol is used over the HPS. Although use of this
protocol significantly reduces the overhead and latency when compared
to IP, on node communication may be more efficiently handled using
the shared memory protocol.
<p>
In addition, the MPI implementation is now layered over the <b>
Low-level Application Programming Interface (LAPI) transport layer
</b>, now available in the current version of the PE. In
contrast to the "reliable byte stream" approach of the previous PE,
LAPI provides a "reliable message" protocol, which uses much less
storage for jobs with a large number of tasks. It is based on an
"active message style" mechanism that provides a one-sided
communications model in which one process initiates an operation and
the completion of that operation does not require any other process to
take a complementary action. In addition, the <b> 64-bit MPI library has
shown improved performance particularly with features such as RDMA, </b>
so it is recommended for users to build there application in 64-bit, even
if they do not have any large memory requirements.
<p>
The <b> Remote Direct Memory Access (RDMA)</b> data transfer feature has also
been made available on the High Performance Switch interconnect. RDMA
allows the network adapter direct access to the user memory of the
application.This allows the network adapter to move data between
computational tasks with minimal CPU involvement, offering greater
potential for overlapping calculation and communication when using
non-blocking communication. RDMA enables large messages to
utilise both network adapters attached to each of the compute nodes
of the p5-575 system without utilising multiple CPUs. This promises a
substantially increased bandwidth for large messages, particularly
for the 64-bit applications. We show below
how RDMA use is enabled by environment variables.
<p>
The parallel operating environment or POE is responsible, among other
things, for managing all communication between the MPI processes whether
on-node or between nodes. The runtime behavior of MPI jobs can be modified
by configuring the many environment variables in POE.
<p>
<center><hr color="#013166" width="50%"></center>
<p>
<a name="shared"><font class="sub">
OpenMP Parallel Programming Environment</font></a>
<p>
Implementation of shared memory parallelization is done through the
creation of user threads, which are mapped to kernel threads by the
operating system.
<p>
OpenMP codes run initially with just one user thread, but a team of threads
is created when the first parallel region is encountered. Upon exiting the
parallel region, only one thread resumes execution of the serial portion,
while the rest consumes CPU time waiting for the next parallel section.
The maximum time a thread spends waiting is regulated by the environment
variable SPINLOOPTIME. After this maximum interval is exceeded,
a thread can either go to sleep or yield its place on the kernel to
another runnable thread, provided that a yield time has been specified as
part of the OpenMP environment. Reactivation of a sleeping thread is more
costly than one that is in a yielded state.
<p>
The behavior of threads can be modified by setting several POE environment
variables. Please check the reference for more details.
<p>
<center><hr color="#013166" width="50%"></center>
<!--
<a name="authorization"><font class="sub">Establishing Authorization </font></a>
<p>
Authorization is set up by the system administrators in the <i>/etc/host.equiv</i>
file. One must set up one's own authentication file using the standard trusted host
file called <b><i>.rhosts</i></b>, which has the following format.
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
longhorn1.tacc.utexas.edu userid
<br>
longhorn2.tacc.utexas.edu userid
<br>
longhorn3.tacc.utexas.edu userid
<br>
longhorn4.tacc.utexas.edu userid
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
The <i>"userid"</i> must be one's actual login name. The
<b><i>.rhosts</i></b> file must be in one's home directory and must be writable by
that user. This is primarily useful when running interactive jobs, since this file
is "looked-up" for processors to run interactive jobs. Users may also want to make it readable by only themselves.
<p>
<hr color="#013166" width="50%">
-->
<p>
<a name="interactive"><font class="section">
Running Interactive Programs</font></a>
<p>
All parallel jobs run under the Parallel Operating Environment (POE). Use the
interactive session only to develop and debug your parallel programs. Use a
command similar to the one given below for interactive jobs.
<p>
<blockquote>
<table>
<tr>
<td><tt><b>% poe your_job job_args -tasks_per_node n -nodes 1 -rmpool 1</b></tt></td>
</tr>
</table>
</blockquote>
<p>
<b><i>n</i></b> is the number of processors on which one wants to run. Interactive runs, which are
queued through the <b>development class</b>, are limited to up to <b>5 processors</b>, for
<b>4 hours</b> and to upto <b>8 GBytes</b> of total memory requirements.
<!-- <p><b>Please note that <a href="http://www.tacc.utexas.edu/resources/allocations/#policies">TACC usage policy</a> does not allow for running of interactive serial jobs (running <i>a.out</i> jobs) on the frontends of our HPC systems. All such jobs must be submitted directly or interactively through the appropriate job scripts. On Power4 'Longhorn' system, use the serial class for that purpose.</b></p>
To be authorized
to run an interactive job be sure that a <b><i> .rhosts </i></b> file is in the
home directory. POE has to be be informed of the processors on which the interactive
job is going to be run. This is done through a <b><i>host.list</i></b> file containing
the names of the nodes on which the job will run. The <b><i>host.list</i></b>
should be in the working directory. One can put it in another directory. However,
the path to the file has to be specified through the environment variable
<b><i>MP_HOSTFILE</i></b> or as a command line argument. The following is a
<b><i>host.list</i></b> file for a job that runs on four processors on a single node.
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
longhorn1.tacc.utexas.edu
<br>
longhorn1.tacc.utexas.edu
<br>
longhorn1.tacc.utexas.edu
<br>
longhorn1.tacc.utexas.edu
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
-->
<hr color="#013166" width="50%">
<p>
<a name="batch"><font class="section">
Running Batch Code using LoadLeveler</font></a>
<p>
<!--Table of Contents-->
<img src="/images/star.gif"><a href="#what" class="link2">
What is LoadLeveler?</a>
<br>
<img src="/images/star.gif"><a href="#configuration" class="link2">
LoadLeveler Batch Resources</a>
<br>
<img src="/images/star.gif"><a href="#simple" class="link2">
Building a Simple Job Command File</a>
<br>
<img src="/images/star.gif"><a href="#advanced" class="link2">
Building an Advanced Job Command File</a>
<br>
<img src="/images/star.gif"><a href="#keywords" class="link2">
LoadLeveler Keywords</a>
<br>
<img src="/images/star.gif"><a href="#submitting" class="link2">
Submitting a Job</a>
<br>
<img src="/images/star.gif"><a href="#managing" class="link2">
Managing a Job</a>
<br>
<img src="/images/star.gif"><a href="#summary" class="link2">
Summary of LoadLeveler Commands</a>
<p>
<hr color="#013166" width="50%" align=center>
<p>
<a name="what"><font class="sub">What is the LoadLeveler?</font></a>
<p>
LoadLeveler is a batch job scheduling software that provides the ability to
submit and manage jobs. It has three interfaces.
<p>
<ul>
<li>Command line interface - provides access to basic job and
administrative functions.</li>
<p>
<li>Graphical user interface (xloadl) - provides the same functionality as
the command line interface; however, the command line interface is
generally found to be more efficient.</li>
<p>
<li>Application programming interface (api) - allows application programs
that are written by users and administrators to interact with the
LoadLeveler environment.</li>
</ul>
<p>
<hr color="#013166" width="50%" align=center>
<p>
<a name="configuration"><font class="sub">
LoadLeveler Batch Resources on Champion</font></a>
<p>
<font class="field">New Resource Limits</font>
<p>
Following the recent upgrade with the IBM Power5 Node(s), the entire
system consists of the eleven eight-way IBM Power5 HPC nodes, and a
login node with 7 processors available for computation. The complete
system consists of<b> 96 Power5 compute processors with an aggregate
memory of 192 GB, 7.2 TB of total disk and theoretical peak
performance of about 730 Gigaflops </b>. <p>
<!-- <b> Note:</b> Semantically, for brevity's sake, we will also refer to
the <b> p690 HPC nodes,p690 Turbo nodes and the p655 nodes as LH16, LH32 and LH4 repectively,</b> in other sections of this document.
<p>
-->
In order to optimize node usage or job throughput on the newly
configured Power5 system, a new resource policy
has been implemented. The
following table summarizes the new resource limits:
<p>
<font class="field">Table I: New resource limits on the TACC IBM Power5 system</font>
<p>
<table border=2 bordercolor="#013166" cellpadding=2 cellspacing=0 align=center>
<tr>
<td class="field"># cpus</td>
<td class="field">memory</td>
<td class="field">walltime</td>
<td class="field">class</td>
</tr>
<tr>
<td align=left valign=top class="text">2-5 procs</td>
<td align=left valign=top class="text">1.8 GB/procs</td>
<td align=left valign=top class="text">4 hrs limit</td>
<td align=left valign=top class="text">development</td>
</tr>
<tr>
<td align=left valign=top class="text">1-2 procs</td>
<td align=left valign=top class="text">1.8 GB/procs</td>
<td align=left valign=top class="text">12 hrs limit</td>
<td align=left valign=top class="text">serial</td>
</tr>
<tr>
<td align=left valign=top class="text">upto 32 procs</td>
<td align=left valign=top class="text">1.8 GB/procs</td>
<td align=left valign=top class="text">24 hrs. limit</td>
<td align=left valign=top class="text">normal</td>
</tr>
</table>
<p>
Users need to understand the logic employed by the filter which parses
the jobscript commands and schedules the job accordingly. Due to the limitation
of the total processor counts, and the homogeneous nature of the Champion system,
relative to the Longhorn system, the filter logic is simplified. If certain user options
are not allowed, then the filter provides the closest options
for the user to make the appropriate choices. The filter logic employs the
following general logic guidelines:
<ul>
<li> User need to specify if they require node usage to be shared or not, unless
all eight processors of the node(s) are in use. Then the
node usage automatically becomes "not shared" or dedicated. The filter
will return the job to the user if this requirement is not fulfillled in
the script. </li>
<p>
<li> All jobs requiring usage of a single node or greater, will require dedicated usage. In
this situation, users are requested to submit processor counts which are
multiples of eight, if feasible. If the user requests, for example 20 processors or 2.5 nodes, they
will be charged for using three nodes. </li>
<p>
<li> In case of users requiring processor counts less than eight, their first option
is to use the development queue. In case of non-availability or if the necessary run time
or the memory requirements exceeds that offered in the development queue, then users
should require four processor count in normal queue for shared usage. Due to limited
resources and for fair useage, users cannot run on processor counts different from
four for sharing usage on a single node. They would need to resubmit the job with
the correct processor count of four. </li>
<p>
<li> Application codes using threads for OpenMP code or a hybrid MPI OpenMPI code
would be allocated usage of a single processor for use of each thread. However, users
must make sure that the total number of processors and threads that are in use, does
not exceed the total number of "(processors) x (nodes)". </li>
<p>
<li> Interactive jobs are currently queued through the development queue class only, and
not allowed to run through any other available queue class. Similarly, interactive serial
jobs are queued through the serial queue class. This policy will be enforced
for fairness to all users. Users who need special requests such as benchmarks, which fall
beyond the perview of the current class queues should make these requests well ahead of
their scheduled runs. </li>
</ul>
<!--
The job may
use all of the nodes memory even though it requires less than 8 processors. In that
situation the user will be charged allocation of all of the eight processors of the node,
for that particular job.
<p>
When running across two or more nodes, users should modify the jobscript to take advantage of the
two HPS Switch adapters on each node. The recommended way of specifying the
Switch adapter is by changing the "#@ network.MPI_LAPI" keyword in the jobscript
to the following:
<p>
<ul>#@ network.MPI_LAPI = sn_all,not_shared,US </ul>
In addition, users may set MP_EUILIB to "US" in order to make use of the
faster User Space (US) implementation of the communication
subsystem library. Jobscript examples which use the HPS Switch can
be found <a href="#building" class="link3">here</a>.
<p> Any other special cases or job runs that do not fall in any of the above queue slots, are
considered special cases and will be considered on a case-by-case basis. Please submit
any such requests through the TACC consulting system.
<p> Since the Power5 system is homogeneous, the number of different
options for job submission is straightforward and simple. (Note: Users
from the previous 'Longhorn' may want to recall the number of
different combinations and options of node combinations with the types
of jobs. These do not exist here).
in nature by the different characteristics
of the Power4 nodes in the configuration, a number of different options of node
combinations
may be possible for a given job description. As an example, an MPI job with 8 tasks and
16 GB of memory may run on one LH16 node (or one p690 HPC node) or two LH4 nodes (or two p655 nodes), if the number of nodes
are not specified. Given such options, the job filter considers the given requirements sufficient and these jobs are then routed
to default node configurations. The default configurations given such options are described
in a table below. However for simplicities sake, we are offering two different kinds
of job scripts -- a simple one for early and first time users and a more detailed
one with all the bells and whistles for advanced users. <p>
<font class="field">New LoadLeveler Classes</font>
<p>
Users should submit jobs to one of four classes: normal, development, and serial
which are described below:
<ol>
<li>The <b><i>normal class</i></b> accepts jobs with processing requirements of 4n, where
n=1,2,3 etc. Walltime and memory limits are based on the job's cpu requirement. Jobs submitted from this class to the LH4's, LH16's and LH32 nodes have a run-time limit of 96 hours. Jobs submitted to the normal class are assigned a priority of
1.0.</li>
<p>
<li>The cpu, memory, and walltime limits for the <b><i>high class</i></b> are identical to those
defined in the normal class. Jobs submitted from this class to the LH4's, LH16's or LH32 nodes have a run-time limit of 96 hours. One important difference between normal and high is the
processing priority. Jobs in the high class are accorded a priority of 2.0 with a
corresponding increase in the SU burn rate.</li>
<p>
<li>The <b><i>development class</i> </b> is intended to have a quick turnover rate, and hence is
suitable for debugging purposes. The resource limits for the development class are small, with
development jobs limited to 4 cpus, 4 hours of walltime, and 12 GB of memory, and they will
only run on the frontend or login node. </li>
<p>
<li> The <b><i>serial class</i> </b> is devoted only to serial jobs and is limited to <b> six </b> jobs at a time running on the frontend. The jobs in this class
are limited to a maximum of 24 hours.</li> <p>
<li>Users are required to make
special arrangements with TACC in order to run in <b><i>dedicated mode </i> </b>
for jobs submitted to the "dedicated" class. </li>
</ol>
<P>
<A NAME="llsubmit">
<P>
<font class="field">The <TT>llsubmit</TT> job filter</font>
<P>
The normal and high class specification is used by an <TT>llsubmit</TT> job filter to determine
the priority of a job. Once determined, the job filter then respecifies the class with a "_lh4,"
"_lh16," or a "_lh32" suffix depending on the set of nodes to which the job is routed. In essence,
a normal or high specification in a job script is reassigned to the
normal_lh4/normal_lh16/normal_lh32 or high_lh4/high_lh16/high_lh32 loadleveler classes,
respectively. The user sees these kinds of nomenclatures for the renaming of the classes to which
their jobs have been submitted.
-->
<!-- Normal1 and high1 jobs are
scheduled
for processing on longhorn5, and are executed concurrently on this
node subject to the availability of cpus and memory. In contrast, each job in
the normal2 and high2 classes has exclusive access to the node of execution
(i.e. only one job at a time is permitted to run on the node(s)),
and may be scheduled on longhorn 6,7, or 8. </A>
-->
<p>
<hr color="#013166" width="50%" align=center>
<p>
<a name="simple"><font class="sub">Building a Simple Job Command File</font></a>
<p>
For all batch submitted jobs, a job script or job command file needs
to be created. This section is for the early or first time users
to the IBM system who need to start with a simple job script described
in this section. Users already experienced with Loadleveler job
scripts should go to the next section where the intricacies
for building more advanced and detailed job scripts are described.
<p>
Some of the general rules and information necessary for building a simple job
script are given below followed by examples of scripts for an MPI and OpenMP job.
<p>
<ol>
<li>Keyword statements, which are case insensitive, begin with #@
followed by LoadLeveler keywords and
statement components; to comment out any statements begin with ##;
</li>
<p>
<li> Common functionality such as shell, initialdir, input, output, job
name etc., are common to all job scripts, and should be taken verbatim from the examples
below and modified according to specific user and job. </li> <p>
<li>Shell command statements can go anywhere except where LoadLeveler
keyword statements are being defined. </li>
<p>
<li> A special keyword statement used in the simple job script is the<b> total </b>number of tasks for MPI jobs and number of threads for OpenMP jobs. Either of these options should be specified by the user in the job script as <b> tasks </b> or <b> threads </b>, depending on whether it is an MPI or an OpenMP job respectively.
</li> <p>
<li> Users should also have an idea of the amount of memory the job consumes. If it
is not specified, the job filter assumes the maximum limit which may not be
sufficient for the job. The job will then be scheduled and may exit with
erroneous statements. The memory statement is another special keyword statement and can be specified as
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>#@ memory = 2</b></tt></td>
</tr>
</table>
</blockquote>
<p>
This implies that the memory per task or thread is 2MB.The total memory is then computed by the job filter by multiplying it with the number of tasks or threads that are specified. Note that generally memory per task or thread should not exceed 2 GB or 2000 MB.
</li> <p>
<li>
Finally, the user should have a rough idea of the runtime. This is for fairness of
how jobs are scheduled. Jobs which take a shorter runtime are scheduled ahead of those
that are longer, all other things being equal, so the user may be penalised in seeing
the expected turnaround time. The specification for the runtime, another special keyword statement, can be done as follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>#@ walltime = 00:10:00 </b></tt></td>
</tr>
</table>
</blockquote>
<p>
This specifies that the wall clock time is 10 minutes.</li><p>
<li> Based on the user providing correct information on all of the above three special keyword statements, the job
filter will then infer the other keyword statements that are necessary for the job to be scheduled. In certain situations, these generated conditions may create contradictory options
for the filter, which is then returned to the user for making the appropriate choice. </li> <p>
<!-- The default options for the different combinations of total tasks/threads, memory and walltime, are available <a href="/resources/user_guides/power4/routing.php" target="_blank">here.</a></li> <p> -->
<li> Another Loadleveler keyword statement
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b># @ job_type = parallel </b></tt></td>
</tr>
</table>
</blockquote>
<p>
also has to be added in addition to the special keyword statements above for all
MPI and OpenMP jobs. Although
this keyword statement is not a special one, it is necessary for the job filter to
know the classification of the job. </li> <p>
<li> Users who belong to multiple projects should specify the project name to charge,
as shown below for the A-abc project:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b># @ account_no = A-abc </b></tt></td>
</tr>
</table>
</blockquote>
<p>
Project names are listed in the accounting report at login.
</li> <p>
<li>The environment variable OMP_NUM_THREADS has to be specified by the user
for OpenMP jobs. All other environment variables will be automatically generated
by the filter for both OpenMP and MPI jobs. </li>
</ol>
<p>
<b>Note: Users should be aware that all of the special keyword statements can ONLY be used in the simple
job scripts. Any combination of the special keyword statements with some of the more
advanced keyword statements will result in incorrect filtering and will result in either the script being returned to the user or an erroneous
job script sent to the scheduler. </b>
<p>
An example of a job command file for <b><i> MPI job </b></i> follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ shell = /bin/csh
<br>
#@ initialdir = /home/login_name/work_dir
<br>
#@ job_name = my_job
<br>
#@ input = /dev/null
<br>
#@ output = $(job_name).o$(jobid)
<br>
#@ error = $(job_name).o$(jobid)
<br>
#@ notify_user = your_email
<br>
#@ job_type = parallel
<br>
#@ environment = COPY_ALL;
<br>
#@ walltime = 01:00:00
<br>
#@ tasks = 8
<br>
#@ memory = 100
<br>
#@ class = normal
<br>
#@ queue
<br>
poe ./a.out
</b></tt></td>
</tr>
</table>
</blockquote>
An example of a job command file for <b><i>OpenMP job </b></i> follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ shell = /bin/csh
<br>
#@ initialdir = /home/login_name/work_dir
<br>
#@ job_name = my_job
<br>
#@ notify_user = your_email
<br>
#@ input = /dev/null
<br>
#@ output = $(job_name).o$(jobid)
<br>
#@ error = $(job_name).o$(jobid)
<br>
#@ job_type = parallel
<br>
#@ environment = COPY_ALL; LL_JOB=TRUE;
<br>
#@ walltime = 01:00:00
<br>
#@ threads = 8
<br>
#@ memory = 200
<br>
#@ class = normal
<br>
#@ queue
<br>
setenv OMP_NUM_THREADS 8
<br>
./a.out
</b></tt></td>
</tr>
</table>
</blockquote>
<hr color="#013166" width="50%" align=center>
<p>
<a name="advanced"><font class="sub">Building an Advanced Job Command File</font></a>
<p>
Every LoadLeveler job must be submitted through a job command file. The job
command file has LoadLeveler statement lines with LoadLeveler keywords that
describe the job. Some of the general rules/information are explained below,
in addition to explaining the more important and complicated LoadLeveler keyword
statements whose understanding is absolutely essential. <p> <b>Note that the special
keywords used for the simple jobscript CANNOT be used for the advanced version
of the job script described in this section.</b>
<p>
Different MPI and OpenMP
jobs require separate job scripts and some examples of the different scenarios
are listed following the explanation of the general rules below.
<!--
<p>
<ul>
<li>Name of the executable,</li>
<p>
<li>Resource requirements,</li>
<p>
<li>Input/ Output files,</li>
<p>
<li>Number of processors needed, and</li>
<p>
<li>Job type - serial or parallel.</li>
</ul>
-->
<p>
<!-- The following rules apply to building a job command file:
<p>-->
<ol>
<li>Keyword statements, which are case insensitive, begin with #@
followed by LoadLeveler keywords and
statement components; to comment out any statements begin with ##;
</li>
<p>
<li>Shell command statements can go anywhere except where LoadLeveler
keyword statements are being defined.</li>
<p>
<li>
<p> The fine grained resource management on node is done by the Work Load
Manager (WLM). <font color="#C32736">The following resource statement
must be included in batch scripts.</font>
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>#@ resources = ConsumableCpus(CPUSpTask)
ConsumableMemory (MEMpTASK)</b></tt></td>
</tr>
</table>
</blockquote>
<p>
<pre>
where MEMpTASK = amount of MEMory required Per TASK,
and CPUSpTASK = number of CPUs required Per TASK.
For OpenMP jobs use the value of OMP_NUM_THREADS for CPUSpTASK.
For MPI jobs use the value of 1 for CPUSpTASK.
</pre>
E.g. if <b><i>ConsumableMemory(1024MB)</i></b> is used with
<b><i>tasks_per_node=4</i></b>, then 4096 Megabytes of memory will
be reserved for the job. For OpenMP programs there is only 1 task,
so you must make MEMpTASK equal to the amount of memory you need
for ALL threads.
[A master thread and its spawned group of threads is a single
task and the ConsumableCPU's argument is the number of CPUs reserved
per (for the) task.]
<p>
<li> Parallel MPI jobs that run across multiple nodes have to specify
the use of the HPS Switch interconnect for off-node
communication. This involves setting <b>MP_EUILIB </b>to <b>US</b> and
<b> MP_EUIDEVICE </b> to sn_all.
A concise way of doing this is to set the #@ network.MPI_LAPI keyword to
the following
<blockquote>
<table border=0>
<tr>
<td><tt><b> #@ network.MPI_LAPI = <device>,<shared|not_shared>,<protocol>
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
<table border=0 align=center valign=middle>
<tr>
<td class="text">where <device> =</td>
<td class="text"><i>sn_all</i></td>
<td class="text">uses dual-plane Switch adapters</td>
</tr>
<!--<tr>
<td class="text"> </td>
<td class="text"><i>css0/css1</i></td>
<td class="text">single Switch adapter per node</td>
</tr> -->
<tr>
<td class="text"> </td>
<td class="text"><i>en1</i></td>
<td class="text">Gigabit ethernet adapter</td>
</tr>
<tr>
<td class="text">where <protocol> =</td>
<td class="text"><i>US</i></td>
<td class="text">to be used by the dual-plane Switch adaptors only</td>
</tr>
<tr>
<td class="text"> </td>
<td class="text"><i>IP</i></td>
<td class="text">can be used by ALL of the interconnects</td>
</tr>
</table>
<p>
<li> A number of <b> environment variables </b>, when used appropriately impact performance.
Some of the environment variables are set as default by the system. Other environment variables
need to be used under certain conditions. Here are a listing of some of them:
<p>
<h4> Default Settings </h4>
<p>
These settings are already set by default and the users need not declare them in the jobscript,
unless they need to be changed.
</p>
<pre>
<b> MP_SHARED_MEMORY=yes</b>
<b> MP_EUIDEVICE=sn_all </b>
<b> MEMORY_AFFINITY=MCM </b>
</pre>
<p>
<h4> Using RDMA </h4>
<p>
Users wanting to use the RDMA feature need to set the following environment in there job scripts, when
running there 64-bit application
<pre>
<tt> <b> MP_USE_BULK_XFER=yes </b> </tt>
<tt><b> MP_BULK_MIN_MSG_SIZE=150K, (Note: this size can be increased upto 2MB.) </b> </tt>
</pre> </p>
<h4> Special Environment Use </h4>
<p>
The following environments can be declared and used in most of the general conditions but not
under all circumstances. These exceptions are pointed out.
<pre>
<tt> <b>MP_SINGLE_THREAD=yes (Except for MPI-I/O or explict MPI threading routines) </b> </tt>
<tt><b>MP_TASK_AFFINITY=MCM (Except OpenMP routines) </b></tt>
</pre> </p>
</li>
<li> All OpenMP jobs, whether "serial" or "parallel", and
hybrid MPI-OpenMP jobs which are
"parallel", are so termed simply from the point of the number of tasks
that are executed. But
each of these tasks spawns multiple threads per process or task. This is
done by the
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ resources = ConsumableCpus(n)..</b>
</tt></td>
</tr>
</table>
</blockquote>
<p>
statement, where <i>n</i>
is an integer between 1-8 for p575 HPC nodes. But this spawning of threads is equivalent
to reserving <i> n</i> processes, not setting OpenMP threads. This is
done by the statement
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
setenv OMP_NUM_THREADS <i>n</i> </b>
</tt></td>
</tr>
</table>
</blockquote>
<p>
<!--
<li>The job command file can be named anything. The convention used by
<b><i>xloadl</i></b> is to add the suffix <i>".cmd"</i> to the filename.</li>
</ol>
<p>-->
<li> For all MPI, OpenMP or a hybrid combination jobs, the job filter
evaluates the number of processors to reserve per node by the following
formula
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
number of processors/node = tasks_per_node x ConsumableCpus(n) </b>
</tt></td>
</tr>
</table>
</blockquote>
<p>
For MPI jobs, <i>n=1</i>, while for OpenMP jobs, <i>n=OMP_NUM_THREADS</i>.
</ol>
An example of a job command file for <b><i> MPI jobs on one node of p575 HPC node</b></i>, follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ shell = /bin/csh
<br>
#@ initialdir = /home/login_name/work_dir
<br>
#@ job_name = my_job
<br>
#@ input = /dev/null
<br>
#@ output = $(job_name).o$(jobid)
<br>
#@ error = $(job_name).o$(jobid)
<br>
#@ job_type = parallel
<br>
#@ environment = COPY_ALL; MP_SINGLE_THREAD=yes;
<br>
#@ resources = ConsumableCpus(1) ConsumableMemory(100mb)
<br>
#@ network.MPI_LAPI = sn_all, not_shared, US
<br>
#@ wall_clock_limit = 01:00:00
<br>
#@ node = 1
<br>
#@ tasks_per_node = 8
<br>
#@ node_usage = not_shared
<br>
#@ notify_user = your_email
<br>
#@ notification = error
<br>
#@ class = normal
<br>
#@ queue
<br>
poe ./a.out
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
An example of a job command file for <b><i> MPI jobs on multiple nodes on p575 nodes </b></i>, follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ shell = /bin/csh
<br>
#@ initialdir = /home/login_name/work_dir
<br>
#@ job_name = my_job
<br>
#@ input = /dev/null
<br>
#@ output = $(job_name).o$(jobid)
<br>
#@ error = $(job_name).o$(jobid)
<br>
#@ job_type = parallel
<br>
#@ environment = COPY_ALL; MP_SINGLE_THREAD=yes;
<br>
#@ node_usage = not_shared
<br>
#@ resources = ConsumableCpus(1) ConsumableMemory(100mb)
<br>
#@ wall_clock_limit = 24:00:00
<br>
#@ network.MPI_LAPI = sn_all, not_shared, US
<br>
#@ node = 4
<br>
#@ tasks_per_node = 8
<br>
#@ notification = never
<br>
#@ class = normal
<br>
#@ queue
<br>
poe ./a.out
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
<b> Note</b>: Comparing the two MPI job scripts for running application codes
on one node and for more than node, the main difference is specifying the appropriate
network/adaptor information. In the latter script, that is contained in the line
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>#@ network.MPI_LAPI = sn_all, not_shared, US</b></tt></td>
</tr>
</table>
</blockquote>
<!--
<p>
See Rule 4 for an explanation of this LoadLeveler statement.
<p>
-->
<!--
Here the first parameter <i><b>csss</i></b> refers to the network adaptor being
used. If you wanted to use the adaptors of the <b> dual-plane Colony or
SP switch2 interconnect,</b>
the adaptor to be used would be <i><b>css0</i></b> or <i><b>css1</i></b> and
<i><b>csss</i></b> for use of both adaptors. If
you prefer to use instead the slower adaptor from the previous <b> Gigabit Ethernet switch
</b> for benchmarking purposes, then you would use instead <i><b>en1</i></b>.
However, note that if you are using the GigE network, then that would only work with
<i><b> ip</i></b> protocol only and not <i><b> us</i></b> protocol, which works only
with the SP switch. Another issue is if the interconnect access is to be <b><i>shared
</b></i> or <b><i> not_shared </b></i>
with other users or not -- this is the second parameter of the above statement.
<p>
-->
An example of a job command file for <b><i>OpenMP job on a single p575 node</b></i>, follows:
<p>
<blockquote>
<table border=0>
<tr>
<td><tt><b>
#@ shell = /bin/csh
<br>
#@ initialdir = /home/login_name/work_dir
<br>
#@ job_name = my_job
<br>
#@ input = /dev/null
<br>
#@ output = $(job_name).o$(jobid)
<br>
#@ error = $(job_name).o$(jobid)
<br>
#@ job_type = parallel
<br>
#@ environment = COPY_ALL; LL_JOB=TRUE;
<br>
#@ resources = ConsumableCpus(8) ConsumableMemory(1000mb)
<br>
#@ network.MPI_LAPI = sn_all, not_shared, US
<br>
#@ wall_clock_limit = 01:00:00
<br>
#@ node = 1
<br>
#@ tasks_per_node = 1
<br>
#@ node_usage = not_shared
<br>
#@ notify_user = your_email
<br>
#@ notification = error
<br>
#@ class = normal
<br>
#@ queue
<br>
setenv OMP_NUM_THREADS 8
<br>
./a.out
</b></tt></td>
</tr>
</table>
</blockquote>
<p>
<b>Note </b>that unlike the MPI jobs, the <b><i>ConsumableCpus(8)</b></i> statement above is
critica]]>