home directory

At login, the system automatically changes to your home directory.
This is the recommended location to store your source codes and build your executables.
The quota limit on home is 200MB
A user's home directory is accessible from the frontend node and any compute node.
Use $HOME to reference your home directory in scripts.
Use cd to change to $HOME.

work directory

Store large files here.
Often users change to this directory in their batch scripts and run their jobs in this file system.
A user's work directory is accessible from the frontend node and any compute node.
The work file system is approximately 30TB
Purge Policy: Files with access times greater than 10 days are purged.
This file system is not backed up.
Use $WORK to reference your work directory in scripts.
Use cdw to change to $WORK.

More on Work -- How to do parallel I/O in the Lustre File System

scratch or temporary directory

This is a directory in a local disk on each node where you can store files and perform local I/O for the duration of a batch job.
Often, in batch jobs it is more efficient to use and store files directly in $WORK (to avoid moving files from scratch at the end of a job).
The scratch file system is approximately 60GB.
Files within the scratch directory on each node are removed immediately after the job terminates.
Use $SCRATCH to reference a temporary directory in scripts.

archive

Store permanent files here for archival storage.
The archive tape capacity is 2.5PB; the disk cache size is 2.5TB.
The access speed is low relative to the work directory.
This file system is NOT NSF mounted (directly accessible) on any node.
Use the rcp command to transfer data to this system.
Use the rsh command to login to archive from any TACC machine.
  e.g.
      rcp ${ARCHIVER}:$ARCHIVE/myfile  $WORK
      rsh archive

Used this system for long-term file storage (tape system); it is not appropriate to use it as a staging area.

More on Archive -- How to use the rcp command

For distributed memory systems, single-program multiple-data (SPMD) and multiple-program multiple-data (MPMD) programming paradigms are used. In the SPMD paradigm, each processor loads the same program image and executes and operates on data in its own address space (different data). This is illustrated in Figure 4. It is the usual mechanism for MPI code: a single executable (a.out in the figure) is available on each node (through a globally accessible file system such as $WORK or $HOME), and launched on each node (through the batch MPI launch command, "ibrun a.out").

In the MPMD paradigm, each processor loads up and executes a different program image and operates on different data sets, as illustrated in Figure 4. This paradigm is often used by researchers who are investigating the parameter space (parameter sweeps) of certain models, and need to launch 10s or hundreds of single processor executions on different data. (This is a special case of MPMD in which the same executable is used, and there is NO MPI communication.) The executables are launched through the same mechanism as SPMD jobs, but a Unix script is used to assign input parameters for the execution command (through the batch MPI launcher, "ibrun script_command"). Details of the batch mechanism for parameter sweeps are described in the Running Programs section.

Figure 4. Distributed Memory Paradigm: Single/Multiple-Program Multiple-Data.

The shared-memory programming model is used on Symmetric Multi- Processor (SMP) nodes, like the TACC Champion Power5 System. Each node on this system contains 8 CPUs with a single 16GB memory subsystem.

The programming paradigm for this memory model is called Parallel Vector Processing (PVP) or Shared-Memory Parallel Programming (SMPP). The latter name is derived from the fact that vectorizable loops are often employed as the primary structure for parallelization. The main point of SMPP computing is that all of the processors in the same node share data in a single memory subsystem, as shown in Figure 5. There is no need for explict messaging between processors as with with MPI coding.

Figure 5. Shared-Memory Parallel Processing.

In the SMPP paradigm either compiler directives (as pragmas in C, and special comments in FORTRAN) or explicit threading calls (e.g. with Pthreads) is employed. The majority of science codes now use OpenMP directives that are understood by most vendor compilers, as well as the GNU compilers.

In cluster systems that have SMP nodes and a high speed interconnect between them, programmers often treat all CPUs within the cluster as having their own local memory. On a node an MPI executable is launched on each CPU and runs within a separate address space. In this way, all CPUs appear as a set of distributed memory machines, even though each node has CPUs that share a single memory subsystem.

In clusters with SMPs, hybrid programming is sometimes employed to take advantage of higher performance at the node-level for certain algorithms that use SMPP (OMP) parallel coding techniques. In hybrid programming, OMP code is executed on the node as a single process with multiple threads (or an OMP library routine is called), while MPI programming is used at the cluster-level for exchanging data between the distributed memories of the nodes.

The number of application that benefit from hybrid programming on dual-processor nodes (e.g. on Lonestar) is very small. The programming and support of hybrid codes is complicated by compiler and platform support of both paradigms. However, with the new multi-core multi-socket commodity systems on the horizon, there may be a resurgence in hybrid programming if these systems provide better enhanced performance with SMPP (OMP) algorithms.

For further information on OpenMP, MPI and on programming models/paradigms, please see the manuals and packages sections of this document.

Compiling and Running Serial Programs

Lonestar programming environment uses the Intel C++ and Intel Fortran compilers as the default compilter system. The following section highlights the important HPC aspects of using the Intel compilers. The Intel compiler commands can be used for both compiling (making ".o" object files) and linking (making an executable from a ".o" object files). The tables below list the syntax for serial and parallel program compilation.

Compiling Serial Programs

Compiler	Program	TypeSuffix	Example
icc	C	.c	icc   [compiler_options] prog.c
icc	C++	.C, .cc, .cpp, .cxx	icc   [compiler_options] prog.cpp
ifort	F77	.f, .for, .ftn	ifort   [compiler_options] prog.f
ifort	F90	.f90, .fpp	ifort [compiler_options] prog.f90

Appropriate program-name suffixes are required for each compiler. By default, the executable name is a.out; and it may be renamed with the -o option. To compile without the link step, use the -c option. The following examples illustrate renaming an executable and the use of two important compiler optimization options:

C	icc    -o flamec.exe -O3 -xP prog.cc
Fortran	ifort -o flamef.exe -O3 -xP prog.f90

Commonly used options may be placed in a icc.cfg or ifc.cfg file for compiling C and Fortran codes, respectively.

To run a serial (non-parallel) executable, simply type the name of the executable on the command line (and hit return).

./a.out {serial code execution}

The relative path expression "./" tells the shell to look in the present working directory for the executable. It is often used to make sure that an executable of the same name in another directory (as determined by the PATH environment variable) is not executed. Also, if the "." is not in the PATH variable it is necessary to use "./" for the shell to find the executable.

A list of all compiler options, their syntax, and a terse explanation, is given when the compiler command is executed with the -help option. Also, man pages are available. To see the help or and man information, execute one of:

icc   -help

ifort -help

man icc

man ifort

Some of the more important options are listed below. Additional documentation, references, and a number of user guides (pdf, html) are available in the /opt/intel/compiler9.x/docs directory.

The Intel 9.1 Compiler Suite

The Intel 9.1 compilers are loaded as the default compilers at login with the intel module. (The 8.1 compilers are available for special porting needs.) The gcc 3.4.4 compiler and module are also available; but we recommend using the Intel suite whenever possible. The 9.1 suite is installed with the EM64T 64-bit standard libraries and will compile programs as 64-bit applications (as the default compiler mode). Any programs compiled on 32-bit systems (including the old Lonestar) need to be recompiled to run natively on Lonestar. Any pre-compiled packages should be EM64T (x86-64) compiled or else errors may occur. Since only 64-bit versions of the MPI libraries have been build on Lonestar, programs compiled in 32-bit mode will not execute MPI code.

The Intel FORTRAN compiler command is ifort (as of Version 8.0) The ifc command is still accepted, but it displays an annoying message about the name obsolescence.

Web accessible Intel manuals are available: Intel 9.1 C++ Compiler Documentation and Intel 9.1 Fotran Compiler Documentation.

Compiling OpenMP Programs

Since each of the PowerEdge blades (nodes) of the Lonestar cluster is a Xeon dual-processor system, applications can use the shared memory programming paradigm "on node". However, because of the limited number of processors in each node, there are rarely any significant performance benefits to using a shared-memory model on the node.

The OpenMP compiler options are listed below for those who do need SMP support on the nodes. For hybrid programming, use the mpi-compiler commands, and include the openmp options.

The "mpicmds" commands support the compilation and execution of parallel MPI programs for specific interconnects and compilers. At login, MPI MVAPICH (mvapich) and Intel 9.1 compiler (intel) modules are loaded to produce the default environment which provide the location to the corresponding mpicmds. The mpicc, mpiCC, mpif77, and mpif90 compiler scripts (wrappers) compile MPI code and automatically link startup and message passing libraries into the executable. The following table lists the compiler wrappers for each language:

Compiling Parallel Programs with MPI

Compiler	Program	TypeSuffix	Example
mpicc	c	.c	mpicc  [compiler_options] prog.c
mpiCC	C++	.cc, .C, .cpp, .cxx	mpiCC  [compiler_options] prog.cc
mpif77	F77	.f, .for, .ftn	mpif77 [compiler_options] prog.f
mpif90	F90	.f90, .fpp	mpif90 [compiler_options] prog.f90

Appropriate program-name suffixes are required for each wrapper. By default, the executable name is a.out; and it may be renamed with the -o option. To compile without the link step, use the -c option. The following examples illustrate renaming an executable and the use of two important compiler optimization options:

C	mpicc  -o prog.exe -O3 -xP prog.cc
Fortran	mpif90 -o prog.exe -O3 -xP prog.f90

Include linker options such as library paths and library names after the program module names, as explained in the Loading Libraries section below. The Running Code section explains how to execute MPI executables in batch scripts and "interactive batch" runs on compute nodes.

We recommend that you use the Intel compiler for optimal code performance. TACC does not support the use of the gcc compiler for production codes on the Lonestar system. For those rare cases when gcc is required, for either a module or the main program, you can specify the gcc compiler with the -cc mpcc option for module requiring gcc. (Since gcc- and Intel-compiled code are binary compatible, you should compile all other modules that don't require gcc with the Intel compiler.) When gcc is used to compile the main program, an additional Intel library is required. The examples below show how to invoke the gcc compiler for the two cases:

mpicc -O3 -xP -c -cc=gcc suba.c
mpicc -O3 -xP mymain.c suba.o
 
mpicc -O3 -xP -c suba.c
mpicc -O3 -xP -cc=gcc -L$ICC_LIB -lirc mymain.c suba.o

Basic Optimization for Serial and Parallel Programming using OpenMP and MPI

The MPI compiler wrappers use the same compilers that are invoked for serial code compilation. So, any of the compiler flags used with the icc command can also be used with mpicc; likewise for ifort and mpif90; and iCC and mpiCC. Below are some of the common serial compiler options with descriptions.

Compiler Options	Description
-03	performs some compile time and memory intensive optimizations in addition to those executed with -O2, but may not improve performance for all programs.
‑vec_report[0|...|5]	control amount of vectorizer diagnostic information:
-xP	generates specialized code to run exclusviely on EM64T processors.
-fast	DO NOT USE -- static load not allowed.
-g -fp	debugging information produced, disable using EBP as general purpose register
-openmp	enable the parallelizer to generate multi-threaded code based on the OpenMP directives
‑openmp_report[0|1|2]	control the OpenMP parallelizer diagnostic level.

Use the -help option with the mpicmds for additional information:

mpicc   -help

mpif90  -help

mpirun -help {use the listed options with the ibrun cmd}

Loading Libraries

Some of the more useful load flags/options are listed below. For a more comprehensive list, consult the ld man page.

• Use the -l loader option to link in a library at load time: e.g.

  ifort prog.f90 -lname

  This links in either the shared library libname.so (default) or the static library libname.a, provided it can be found in ldd's library search path or the LD_LIBRARY_PATH environment variable paths.

• To explicitly include a library directory, use the -L option, e.g.

  ifort prog.f -L/mydirectory/lib -lname

  In the above example, the user's libname.a library is not in the default search path, so the "-L" option is specified to point to the libname.a directory.

  Many of the modules for applications and libraries, such as the mkl library module provide environment variables for compiling and linking commands. Execute module help module_name for a description, listing and use cases for the assigned environment variables. The following example illustrates their use for the mkl library:

  mpicc	-Wl,-rpath,$TACC_MKL_LIB -I$TACC_MKL_INC mkl_test.c \
  	-L$TACC_MKL_LIB -lmkl_em64t

  Here, the module supplied variables TACC_MKL_LIB and TACC_MKL_INC contain the MKL library and header library directory paths, respectively. The loader option -Wl specifies that the $TACC_MKL_LIB directory should be included in the binary executable. This allows the run-time dynamic loader to determine the location of shared libraries directly from the executable instead of the LD_LIBRARY path or the LDD dynamic cache of bindings between shared libraries and directory paths. (This avoids having to set the LD_LIBRARY path ("manually" or through a module command) before running the executables.

  Note: For FORTRAN codes that used utilities such a getarg it was necessary to include the compatibility library libPEPCF90.a with the "-Vaxlib" option when using the Intel compiler. As of version 9.1, this option is no longer required.