By Avrel Seale

September 28, 2006, was a good day to be Jay Boisseau. That was the day it was announced that the National Science Foundation would make a five-year, $59 million award to UT's Texas Advanced Computing Center (TACC) to acquire, operate, and support a high-performance computing system that will provide unprecedented computational power to the nation's research scientists and engineers. It was the largest NSF award ever to UT Austin.

The UT project team is led by Boisseau, director of TACC, and includes leading researchers from TACC and the Institute for Computational Engineering & Sciences (ICES). UT Austin, in collaboration with Sun Microsystems, Arizona State University, and Cornell University, submitted the proposal in response to the NSF's inaugural competition of the High Performance Computing System Acquisition Program. The award covers the acquisition and deployment of the new Sun system and four years of operations and support to the national community to enhance leading research programs.

TACC will be the lead partner, with assistance from ICES, ASU, and Cornell in the areas of applications optimization, large-scale data management, software tools evaluation and testing, and user training and education.

Supercomputing is the unsung hero working behind the scenes, crunching numbers for virtually every major scientific study today. It remains in the background, only occasionally stepping forward to make headlines for itself with breakthroughs in the science of computation itself.

But let's start at the beginning.

What is high-performance computing?

Most scientific and engineering research is dependent on computation. "This is clearly a change from the many centuries of scientific and engineering research since classical Greece to the beginnings of the Western scientific method," says Boisseau. The object of science has always been to discover the way the universe works in a way that is mathematically predictable. "You didn't have a true understanding until you had the ability to predict," says Boisseau. For centuries, those discoveries were made through a combination of direct observation and theory.

Book publishing provided a tremendous advance for science because scientists suddenly were able to build upon each other's research. But as knowledge grew exponentially, we soon reached the point where our ability to understand the world required solving ever-greater numbers of mathematical equations.

With computers really coming of age about a half-century ago, scientists had a new instrument at their disposal.

Whereas previously, their tools had been specific to understanding their domain of interest (a biologist's microscope, a chemist's Bunsen burner), the computer was a general instrument that allowed them to test their best understanding of the physics and mathematics against experimental observations of nature. If they matched, the code could make a prediction about a new scenario that might be difficult, dangerous, expensive, or even impossible to observe in a laboratory.

"Just trying to understand how one ball rolls down a plane doesn't require many computations," explains Boisseau.

"Trying to understand how 100 billion balls of hot gas orbit around each other in a galaxy - that's an impossible amount of computations for anyone to try to pull off by hand." This is the essence of high-performance computation. It allows scientists to test theories and analyze vast volumes of experimental data generated by modern scientific instruments, such as the very high-energy particle accelerators in the United States and Europe. It makes it possible for researchers to conduct experiments that would otherwise be impossible: studying the dynamics of the Earth's climate in the distant past, investigating how the universe developed, or discovering how complex biological molecules mediate the processes that sustain life.

About 30 years ago, there was a very specific meaning to the term "supercomputer." All of the other computers in existence were very low power, and there was a tremendous gap between them and very high-end computers. "There was no spectrum of computer power. Most computers were not much more powerful than the chips in greeting cards that play 'Happy Birthday.' But a couple of companies made special computers that were far more powerful than the 'regular' computers of that era - though only about as powerful as a PC of 10 years ago," says Boisseau.

High-end computing made new kinds of research possible.

A lot of their earliest uses were trying to predict weather by solving dynamics. Now, in industry, high-performance computing is used in everything from aircraft design and improvement of automobile crash-worthiness to the creation of breath-taking animations in cinema.

Supercomputers, known in the field as "HPC systems," are enabling researchers to address important problems in nearly all fields of science. From understanding the 3-D structure and function of proteins to better predicting severe weather, supercomputers have become indispensable to life sciences,geosciences, social sciences, and engineering, producing results that have direct bearing on quality of life.

Moreover, supercomputers are required for basic research across disciplines, from understanding the synthesis of all heavy elements via supernova explosions to mapping the evolutionary history of all organisms throughout the history of life on Earth.

What is Ranger?

The new Sun HPC system at TACC, nicknamed "Ranger" in the center's tradition of naming its supercomputers for Texas icons, soon will become the most powerful computer in the TeraGrid, the National Science Foundation-sponsored network of advanced computers used for science and engineering research and education nationwide.

TACC is partnering with Sun Microsystems to deploy a supercomputer system specifically developed to support very large science and engineering projects. This system will be Sun's largest installation.

In its final configuration in 2007, Ranger will have a peak performance in excess of 500 trillion floating-point operations per second (teraflops), making it one of the most powerful supercomputer systems in the world.

Supercomputers mark a return to the days when it took an entire building to house a machine. The Ranger system will rest in about 100 racks and will occupy about 4,500 square feet (including the space around it to provide cool air) in TACC's new building, completed in January on UT's Pickle Research Campus in north Austin.

Ranger includes 15,700 of AMD's forthcoming quad-core processors that have four processing cores each, to make more than 63,000 processor cores, each slightly better than what most people have in their laptop. For those keeping score at home, it also has 125 terabytes of memory, 100,000 times as much as the average home PC. It also will provide: 1.7 quadrillion bytes (petabytes) of disk storage; the system is based on Sun Fire x64 (x86, 64-bit) servers, and has Sun StorageTek disk and tape storage technologies.

Ranger is planned to be up and running December 1.

The Evolution of TACC

So how did TACC become one of the biggest players in academic computing? The story of TACC begins in 1986, when Hans Mark, then-chancellor of the UT System, expressed his vision that all scientific labs should have access to supercomputing.

"All fields of science must still do observational science," says Boisseau, "but he knew that progress in all fields of science would depend on the people in those fields being able to conduct very large-scale simulations and analyses." Mark set up a UT System supercomputing center at the Pickle Campus, a facility that ran for several years. Though funding was cut in a subsequent administration, the center continued to operate until it became one of the mid-range sites in 1997 in a larger network created by the NSF.

The current story began in 2000, when the center underwent an external review. The review committee told the University it wasn't really supporting TACC properly.

Essentially Hans Mark's vision was too clear, says Boisseau.

"It was 20/10 instead of 20/20. But it was all coming around, and the review panel concluded, rightly, that if you wanted world leadership in science and engineering, you better have access to the best computational resources and the experts to help you use these resources. That gives you the scientific discovery advantage and a tremendous competitive advantage in pursuing funding for the next level of scientific discovery." The panel suggested that the center should have a very different mission from the rest of the IT department. The center was moved under Juan Sanchez, vice president for research, who recruited Boisseau to be the center's director. Boisseau remembers it as a time of "tremendous building opportunity," diplomatic code for, "there wasn't much here," relatively speaking: a dozen talented staffers, but older, smaller-scale resources, and not much funding, and no complementary inhouse R&D activities. "It was definitely a mid-range center at best, barely on the national radar," he says.

"We started turning it from a research facility into a research center. We provided resources and services, but we also began to conduct research needed to develop software techniques that would augment these." When Boisseau came in 2001, there were 15 staffers at TACC. Five years later, there are more than 60 staff and student workers, a number expected to double to 120 over the next four years (another factor necessitating TACC's new building).

The TACC staff hails from a variety of backgrounds. "We're kind of hybrids, kind of 'tweeners," says Boisseau. "There isn't an academic department that trains you to do exactly what we do. And yet it's a crucial niche to fill for scientific discovery." This rapidly growing team has been deploying ever-largerscale resources and increasing its portfolio of support activities.

Now it has three main functions: resources and services, research and development, and education and outreach.

What is the TeraGrid?

The TeraGrid project was launched by the National Science Foundation in August 2001 with $53 million in funding to four sites. The goal - to create a national network of supercomputers.

In 2003, as part of its second expansion, the NSF invited UT's TACC to join the TeraGrid. The network took centers that previously were islands of resources and expertise and transformed them into partnerships through cyberinfrastructure.

"NSF now wants many centers, all being very integrated. Cyberinfrastructure is the buzz word that really means integrating all the high-end computing, storage, visualization resources, data collection, displays, and high-speed networking," says Boisseau. "The vision is to make it a more seamless fabric, so a researcher has a virtual laboratory with access to data, storage, simulations." Between 2004 and today, TACC continued to increase its importance to the TeraGrid, demonstrating leadership in various areas.

The NSF was integrating its resources better, but federal funding for academic facilities had begun to stall, while the departments of defense and energy and NASA were getting one or more high-end computing centers each.

When the NSF sent out a new set of solicitations with much larger-scale HPC systems, The University of Texas partnered with Sun Microsystems for its bid and competed against all of the other major NSF centers, several other major research universities, and some of the major DOE research labs.

It was a closed process, but the word on the street was that there were 13 other bidders. The NSF had the opportunity to award two $15 million awards or one $30 million, and they chose to award all $30 million to the UT/Sun team, which meant another $29 million to support operations. "There's nothing currently in operation in the world like this system.

It's hard to say what else might happen in the world, in some other country, or in a DOE lab, but in all likelihood, it will be the most powerful general-purpose computing platform in the world at the end of 2007.

"It'll make TACC, by one dimension, the No. 1 academic supercomputing center in the U.S. After all these years of being sort of behind the big boys, this puts us at No. 1," says Boisseau. "Centers will always leapfrog one another because whoever gets the latest system has a chance to leapfrog whomever was on top the year before that." But this will enable researchers to go places they've never gone before. "What we tend to do is take the maximum capacity of a system we have at the time and put as much of the physics and mathematics as you can fit on that system.

Then you scale the problem size to one that fits on the system.

With a bigger system you can solve larger problems, get much finer resolution, and run simulations for much longer timeframes, so you produce much more realistic results and also incorporate physics that may have to be left out as approximations in smaller-scale simulations." There won't be much need for approximations with Ranger: This system will have more than 500 trillion floatingpoint operations per second, "a tremendous boost in power." This will enable:

-- Analysis of subterranean structure and whether that structure includes large deposits of oil. This will enable reservoir modeling, understanding where to place various valves to maximize production.

-- Understanding how proteins fold on themselves.

"Understanding the human genome gives you a base to work from but it doesn't tell you how the proteins function," says Boisseau. This will help catalog that crucial knowledge.

-- Trying to understand the fundamental properties of nature, which will allow us to create lighter, stronger, more cost-effective materials.

-- Evolution of the universe on its larger scales, and how stars turn into black holes.

-- Better understanding of climate change, which has immense societal significance. Ranger will better calculate the conveyance of thermal energy between land masses, vegetation, and oceans and help calculate the location and extent of ice cap and glacier melt.

Every person today has benefited from advanced computing.

If you drive a vehicle, it was designed with supercomputing that calculated its safety performance long before it was ever built and tested, along with its fuel economy and performance.

There's no weather forecast more than five minutes out that doesn't use advanced computing.

After Hurricane Katrina hit, TACC became the warehouse for a lot of the satellite data collected by UT's Center for Space Research that aided in the disaster relief effort. But CSR's more important work, arguably, is in using computing to predict the levels of inundation in a given area depending on the level of the storm. (See "First Warning" sidebar.) Biology, chemistry, aerospace engineering, mechanical engineering, petroleum engineering - every science and engineering department will have use for it. "We even have a user in finance," says Boisseau. "We expect to see an increasing amount of use in social science for large-scale data analysis.

As we get even more powerful, we may see new models in economics simulations that are mostly social science but that might include effects from climate change."

The line starts back here (Who can use Ranger?)

TACC allocates computing time to the University, to the UT System, and nationally through the TeraGrid, depending on who has ponied up the dough to build these massive systems.

Lonestar, TACC's star of the moment, is allocated more to UT Austin and the UT System than it is to the national community. "That's a tremendous advantage to researchers at UT Austin. They have an easier time getting access on Lonestar than the rest of the national community, and it's the second most powerful academic computing system in the U.S. right now," Boisseau says.

Ranger will be nine times as powerful as Lonestar, no slouch itself as a 55-teraflop cluster. But these systems are so cutting-edge that researchers have to do the academic equivalent of an elaborate mating dance to gain access. Researchers requesting computation time must specify whether they know how to use the system, what techniques they are going to use, and what they have done so far that indicates their project would be successful. "In some ways it's like requesting time on the Hubble Space Telescope. We take requests from all of those communities and rank them," says Boisseau.

Ultimately, it is a volunteer committee of experts at the NSF that makes allocations on a quarterly basis.

But unlike the Hubble Space Telescope, which essentially has to do one job at a time, a system of this scale can run a number of applications at any given time and has a queuing structure in which it can move another job onto available processors.

This is such a large instrument that it would be a waste to use it merely as a capacity system - to have hundreds of projects, each with five or 10 users. "It's a unique resource with tremendous power to solve problems that couldn't be solved on other systems. So priority will be to allocate time on it to the projects that need a very large system." TACC expects there will be around 20 large-scale projects on it when it is fully operational, with perhaps 20 small-scale projects as well.

But under the NSF agreement, they will reserve up to 5 percent of the time for Texas higher education institutions - including not just major research universities but also health science institutions, junior colleges, minority-serving institutions, and small colleges.

They also will reserve another 5 percent for industrial partners - companies interested in exploring how high-end computing could create breakthroughs in the services and technologies they provide. Companies could use the system for research that would normally be cost prohibitive and might yield uncertain results. "The stock ticker shows their stock price every five minutes, so unfortunately we see a lack of investment in long-term research, but this might provide an opportunity by leveraging our resources and enabling them to do different things," says Boisseau.

The economics of cutting edge research such as Ranger will perform are high-stakes. "If you just hand over a very expensive but complex instrument to researchers who do not yet have the expertise to use it effectively then the investment will not be fully realized. You need staff specifically trained with such systems to work with and support researchers to see the full scientific impact." Boisseau explains that the project only has a four-year life cycle, and with the better part of a year to get up and running, it's basically a five-year project. When one factors in the cost of powering, cooling, and a brand new building to house it, instead of a $60 million project, it might be closer to $70-$75 million. Divide that by the four years it will be in use, and it comes to almost $2 million a month. "You really can't afford to not use this thing. A day of downtime equates to tens of thousands of dollars lost. It's a very expensive instrument, so you want to get the maximum impact out of it. You want it running as much as possible and to know that you're using it as effectively as you can."

Texas Pride

A large number of Texas Exes work at TACC. "There's a lot of UT spirit here."

There are surely more than 45 UT degrees at work at the center," says Boisseau. "We've brought people in from all over the country, but there's a strong feeling of pride with being associated with The University of Texas at Austin. We couldn't have built this center just anywhere. It was a major rebuilding task." But Boisseau says that when you take a world-class university in one of the best cities in the world to live, add a huge technology base and a state with penchant for being No. 1, you've got the ingredients for success. "I didn't come here to be top 10; I came here to be No. 1. You're not going to be No. 1 in a place that's not really committed to that. At UT, you can get close enough that you can get the support needed, people will rally around, and, with the right vision, commitment, passion, and talent, you can do it here. It's not possible to do it everywhere, but it's definitely possible at The University of Texas." Another initiative of TACC is to encourage students even as young as grade school to become engaged and interested in math and science again.

"We want to be part of this national effort. We need for children to grow up and take these careers seriously," says Faith Singer-Villalobos, TACC's public relations coordinator. "Our country is losing a lot of talent in that area." With the TACC team poised to launch Ranger in October, the University, and the nation, can expect mind-blowing advances in virtually every area of science and engineering.

Prepare to boot.

For more information about TACC, readers can visit http://www.tacc.utexas.edu or contact Faith Singer-Villalobos, Public Relations Coordinator, at faith@tacc.utexas.edu.

This article appeared in the March/April 2007 edition of Alcalde Magazine.