$WORK Directories

The work file system is configured with fast disks on TACC machines and should, therefore, be used when I/O performance significantly affects program performance. Work can also be used to store large files temporarily.

The files in work ARE NOT backed up and are temporary. Files that are corrupted or accidentally removed are not recoverable. Files that are not accessed within 10 days are removed. (Each morning a "scrubber" program evaluates access times for every file and removes outdated files. The scrubber does not remove any files if the user has a running batch job. Reminder: for permanent storage use the TACC data archive.

PLEASE NOTE: TACC staff may delete files from work if the work file system becomes full and directories consume an inordinately large amount of disk space, even if files are less than 10 days old. The top 10 largest directories are targeted for pruning first. A full work file system inhibits use of the file system for ALL users.

Lustre is a parallel file system. It is accessible from any node on the system, and looks and feels line a normal Unix file system. For all users this means that their application's I/O (open,read,write, and close operation) and Unix file management commands (ls, mkdir, etc.) behave as usual. For users that handle large amounts of data, the Lustre architecture provides parallel access to multiple files, or parallel access to a single file with an application. The latter is supported through MPI-IO, an MPI implementation for performing parallel I/O within an applications. Users who are interested in employing parallel I/O in there applications should consult
The Lustre file system appears as a "none" file system mounted on /mnt/lustre in the df command, as shown below:


lonestar% df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0              40313912  11252556  27013476  30% /
/dev/sda1               101086     31284     64583  33% /boot
/dev/sdb1               101086     27904     67963  30% /boot.back
none                   2019264         0   2019264   0% /dev/shm
/dev/md1              22074236   8950496  12002412  43% /export
lonestar-home:/export/home

The Lustre file system consists of node clients (the compute nodes), metadata servers (not accessible by the user), and Object Storage Targets (OSTs). The OSTs are, in essence, a set of parallel IO servers where files are written. (For those familiar with NSF, think of the Lustre file system as multiple NSF servers supporting simultaneous access to files.) The architecture is illustrated in Figure 1. The user sees these serves as a single (mounted) file system.

The most important HPC attribute about the Lustre file system is that it supports user-level file striping across the OST (I/O servers), and the striping can be set at the file and directory level. Striping is invoked by the user to increase the read/write performance of large files (over 100 MB). The bandwidth is enhanced by storing the file data is chunks (in round robin fashion) in multiple OST (servers). Hence, large-sized I/O requests will simultaneous read/write to multiple OST (servers).

When accessing a file from a single-processor application or untility, striping across OST will enhance performance up to the limit of what the CPU can handle. Hence, striping works well up to about 2 or 3 OST. However, within MPI applications that use parallel IO (MPI-IO), multiple processors accessing multiple OST's provide very large I/O bandwidths. Using all 16 available OST will provide maximum performance; but remember, many other uses are sharing the system. ------------