The TeraGrid has been in the news again as a new infusion of funding fuels an ambitious program to broaden and deepen the use of grid resources in scientific research. Begun four years ago with funding from the National Science Foundation (NSF), the TeraGrid is a visionary project, a partnership of people and resources that provides comprehensive cyberinfrastructure to enable discovery in U.S. science and engineering research.

The fresh round of funding from NSF brings $10 million to The University of Texas at Austin for the Texas Advanced Computing Center to enhance its participation in the TeraGrid. “TACC is proud to represent the university in working with some of the nation’s leading institutions to develop, operate, and evolve the TeraGrid,” says Dr. Jay Boisseau, TACC director and the principal investigator for the university on the project.

Through high-performance network connections, grid-computing software, and coordinated services, the TeraGrid integrates a distributed set of the highest capability computational, data management, and visualization resources to make U.S. researchers more productive. With Science Gateway collaborations, links to large-scale instruments like telescopes and particle accelerators, and an array of education and outreach programs, the TeraGrid also broadens the scientific communities that can make good use of its resources. TeraGrid director Charlie Catlett of Argonne National Laboratory and the University of Chicago says that the project "will be a persistent infrastructure whose capabilities deepen and evolve. As scientists come to depend upon it, the nature of their research and the questions they can ask will also broaden and deepen."

TACC is one of eight U.S. supercomputing centers that house the TeraGrid’s computational, storage, software, and instrument and visualization resources. The other resource providers are the University of Chicago/Argonne National Laboratory; Indiana University; the National Center for Supercomputing Applications; Oak Ridge National Laboratory; Pittsburgh Supercomputing Center; Purdue University; and the San Diego Supercomputer Center. Currently about 1000 investigators are using the TeraGrid; in five years, Catlett hopes that figure will rise by a factor of 5 to 10.

TACC Contributions to the TeraGrid

"TACC's role in the TeraGrid is very broad," says Boisseau. "We bring into the mix a blend of commodity and leading-edge machinery,knowledge resources, software technologies, and expertise that is unique in the world of supercomputing."

High-Performance Computing: "TeraGrid users can take advantage of our Cray/Dell Linux cluster, called Lonestar," says Chris Hempel, TACC associate director for resources and services and site lead for the TeraGrid. "Lonestar is a massively parallel machine with 1024 compute processors," he says. The peak performance of the machine is 6.2 teraflops (trillions of floating-point operations per second), and the total memory capacity of the machine is 1 terabyte (1 TB, one trillion bytes). Local storage and global work space are 18 TB and 3 TB, respectively. An IBRIX Global File System is available via high-speed access from every node, with a total capacity of 6.2 TB. A Myrinet-2000 switch fabric, employing PCI-X interfaces, interconnects all compute nodes with a point-to-point, unidirectional bandwidth of 2 gigabits per second.

Researchers at The University of Texas at Austin and around the country, via TeraGrid, have been using Lonestar for large-scale calculations in physics and geophysics (including final gravity-field calculations for one of NASA's most exacting earth science missions, being conducted by the university's Center for Space Research), biochemistry (including massive projects in RNA sequence/structure relations and phylogenetics), chemistry (including intensive quantum dynamics calculations for chemical reactions), engineering (including first-principles calculations of turbulent fluid behavior), and other disciplines. "As one of the many powerful high-end compute resources available via the TeraGrid," Hempel says, "Lonestar has been finely tuned to assist researchers in getting optimum turnaround and efficiency."

Remote and Collaborative Visualization Capability: "This machine was designed with remote visualization capability in mind," Boisseau says of Maverick, a Sun E25K platform available to university and TeraGrid researchers. Maverick has 64 dual-core 1.05 GHz processors and 512 gigabytes of shared memory. A special feature of Maverick is the additional presence of 16 graphics processor cards exactly like the systems that power fast video games. "We give large-memory and interactive jobs priority on Maverick," says Gregory S. Johnson of TACC, manager of TACC's Visualization and Data Analysis group. Maverick can also be used from remote locations to drive high-end display equipment, especially important for collaborative research. "This makes it one of the most versatile platforms on the TeraGrid for scientific visualization," Johnson says.

Database Hosting: TACC is making data collections from four major university research centers available to users of the TeraGrid. The data come from the Bureau of Economic Geology, Center for Research in Water Resources, Center for Space Research, and the High Resolution X-ray Computed Tomography Facility. TeraGrid users can access high-resolution digital terrain data, worldwide hydrological data, global gravity data, and computed tomography of living and fossil specimens. These data sets are invaluable research tools for scientists in environmental, geological, climate, and biological research programs.

Science Gateway : Under the joint direction of Dr. Gordon L. Wells of the university's Center for Space Research and Dr. David Maidment of the Center for Research in Water Resources (CRWR), geologists and hydrologists will be able to develop facilities to anticipate and model the effects of flash floods, storm surges, and tsunamis, via the TeraGrid Flood Modeling Science Gateway. Like the other TeraGrid science gateways, the Flood Modeling system will unite the data and tools used by an entire community of scientists and emergency managers. Data used through the gateway will include LiDAR land elevations and NEXRAD precipitation data, as well as satellite data on storms and hurricanes. Models developed by CRWR will enable flood prediction in real time for wide areas subject to sudden flooding. Tomislav Urban, interim manager of TACC's Data and Information Services group, is the liaison for the Flood Modeling Science Gateway.

Software Technology Development: TACC's expertise in the development of user-friendly interfaces for high-end resource monitoring and interaction is being put to work in the development of the TeraGrid User Portal. The portal is a tool for users that will offer grid information and interactivity through a standard web browser. The initial goals for the TeraGrid portal are to simplify account creation for users by hiding and automating grid credential creation; to provide a web interface for viewing allocation usage on all TeraGrid resources; and to enable access to dynamic grid information about the resources, including load, status, and job queue tables. Other features, such as interfaces for data management and job submission, will be added incrementally throughout the life of the project. Software developer Eric Roberts of TACC leads the TeraGrid portals working group and also participates in and contributes to the science gateways effort.

A further software technology development contributed to the TeraGrid is GridShell, a facility for creating a common environment for managing scientific jobs on all the different resources available on the TeraGrid. GridShell was developed by Dr. Edward Walker of TACC in collaboration with other TACC researchers and scientists at the University of Wisconsin, Caltech, and the Pittsburgh Supercomputing Center. GridShell enables transparent access to grid resources via software agents, and it has been used very successfully in the TACC UT Grid project to schedule computational runs.

TeraGrid: A Vision and a Promise

"The contributions TACC is making to the TeraGrid, listed above, are central results of the research programs in which TACC has been active for several years. A pure facility without science-driven research and technology would not have been able to tackle these tasks, and TACC has taken a visionary path here in service to the sciences," says Dr. Kelly Gaither, TACC associate director for research and development. "We expect to expand our research and development programs so we can continue meeting the challenge of building a national infrastructure for the computational sciences."

Ultimately, Boisseau says, "the TeraGrid is defining a new vision of the potential offered by integrating computing resources, visualization systems, data collections, and instruments into a 'cyberinfrastructure' that enhances the capabilities of researchers. Scientists from The University of Texas at Austin and elsewhere around the country will tackle new challenges, scaling up applications in ways that would have been impossible before now. The TeraGrid serves as an exemplar of what is possible, and TACC is playing a leading role in this transformation of scientific capabilities."