Explosive Science
University of Utah researchers test new blast simulation tool on Ranger
Story Update – Sept. 11, 2009:
The C-SAFE team is pleased to announce the release version 1.1.0 of the Uintah software suite. The Uintah software suite is a set of libraries and applications for simulating and analyzing complex chemical and physical reactions. Uintah's underlying technologies have led to novel techniques for understanding large pool eddy fires as well as new methods for simulating fluid-structure interactions. The software is general purpose in nature and the breadth of simulation domains continues to grow beyond the original focus of the C-SAFE initiative. To learn more about Uintah, including downloading the software, please visit www.uintah.utah.edu.
Story Highlights:
|
On August 11, 2005, a truck carrying 35,500 pounds of booster explosives crashed along a Utah highway, blowing a crater in the road 40 feet wide by 25 feet deep.
Like the World Trade Center collapse, the scale of the Utah blast was far larger than anyone expected, partly because the dynamics of explosions — the volatility of fire and fuel — are difficult, expensive and dangerous to gauge experimentally.
A steel container rupturing (red spheres) due to deflagrating plastic bonded explosive (blue spheres). The gaseous product of reaction is shown escaping through the gaps in the container. [credit: Jim Guilkey, J. Davison de St. Germain, Todd Harman, Justin Luitjens, John Schmidt] |
A new computational simulation tool seeks to rectify that. Developed by the Center for the Simulation of Accidental Fires and Explosions (C-SAFE) at the University of Utah and sponsored by the Department of Energy and the National Science Foundation (NSF), the Uintah software combines the most advanced mechanical, chemical and physical models into a novel computational framework, to predict how different containers and devices will react when they combust.
“The C-SAFE project started in 1997 with the goal of pushing the state-of-the-art in scientific simulation technology,” explained Charles Wight, Professor of Chemistry and Deputy Director of C-SAFE. “We wanted to demonstrate how you can bring a group of people together with expertise in various areas — computer science, mechanical engineering, chemical engineering and chemistry — to make a code that reflects the underlying fundamentals of an energetic device embedded in a fire, and can predict behavior that you can’t do experiments for.”
Using the Ranger supercomputer at the Texas Advanced Computing Center (TACC), the Uintah researchers have been testing their software at an unprecedented scale. Their goal is to make Uintah available to the broad scientific community through TeraGrid, the nationwide network of academic HPC centers, sponsored by the NSF Office of Cyberinfrastructure, which provides scientists and researchers access to large-scale computing, networking, data-analysis and visualization resources and expertise.
“The primary aim of the project is to get the Uintah software to a point where it can be used on the TeraGrid by general users who would like to solve problems with the tool,” said Martin Berzins, Director and Professor at the School of Computing at the University of Utah. “A key part of that is making sure that the software is capable of making full use of machines like Ranger.”
By scaling from 4,000 to 32,000 processors (soon to be 60,000), the C-SAFE developers are producing more highly resolved simulations than ever before, revealing small-scale effects within the combustion process and proving the robustness of the code for large-scale simulations.
Explosions are particularly challenging to simulate because they involve many different branches of science (encompassing chemistry, engineering and physics) and must model interactions on a range of dimensions — from the molecular level to meters in length — and durations — from microseconds to minutes.
“When you talk about the combustion of an explosion, you have to deal with a variety of different length scales and time scales,” explained Wight. “In order to fit that into a single end-to-end simulation, you need many thousands of processors to do the problem properly.”
“We’re right on the bleeding edge of technology here, and there are always challenges in terms of whether all the components actually work. However, the things that we learn at each stage definitely help us move on to the next one and with every iteration of that process, the code gets better and we learn how to scale the code on these very large machines.” Martin Berzins, Director and Professor at the School of Computing at the University of Utah |
Even Ranger's 62,000 processors can’t model these explosions at the finest resolution, Berzins said, noting that “the number of processors that we could actually use is unbounded, because we could keep putting detail into these models to get better accuracy and reliability.”
Expanding the code to run on Ranger's massive parallel system is not a trivial problem. Few algorithms can process efficiently at the scale the C-SAFE researchers are attempting, and novel aspects of their code made it a particularly difficult process.
Uintah uses a common component architecture (CCA) that lets scientists write serial algorithms relating to their individual disciplines without having to worry about the parallel computing aspects, like load balancing or communication. “This allows the application developers to focus on what we know best and leave the computer science to the other part of the framework,” said Todd Harman, one of the project’s senior developers.
The software also uses asynchronous communication, whereby messages are sent and received in a less ordered way than usual. This feature will ultimately allow the code to scale efficiently to millions of processors, its creators believe, but proved difficult to implement at the highest processor counts.
“Anytime you have a code that pushes the boundaries of high performance computing, different parts of the code are going to come into play as bottlenecks and you’ll find bugs that no one knew about,” Wight said. “We’ve broken just about every compiler that we’ve ever used, every MPI framework that we’ve ever used, just by pushing to larger and larger problems.”
With the help of TACC’s systems experts, the Uintah team has overcome these obstacles, proving that their combustion framework works well at the largest supercomputing centers as well as on single processors.
“The Uintah team has made incredible progress in identifying algorithmic and implementation-specific details which effect scaling at 32K distributed tasks and beyond to achieve impressive results in combustion simulation,” says Karl W. Schulz, an Associate Director at TACC.
Visualization of a large JP8 pool fire in cross flow that is heating a suspended cylindrical container. [credit: Jeremy Thornock, Jennifer Spinti] |
“We’re right on the bleeding edge of technology here, and there are always challenges in terms of whether all the components actually work,” said Berzins. “However, the things that we learn at each stage definitely help us move on to the next one and with every iteration of that process, the code gets better and we learn how to scale the code on these very large machines.”
One of the main assets of Uintah is its flexibility, giving it the capacity to simulate phenomena beyond fires and explosions. On Ranger, the group has explored applications ranging from the modeling of vocal chords to the simulation of granular matter around oil wells to the development of novel computer cooling systems — problems they, or other researchers, will continue to study using Uintah.
The software is particularly well suited to problems that are difficult to test experimentally, whether they involve massive hazards, high costs or impossible to recreate conditions.
“Recently, there have been a couple of well-publicized fires on a space station,” said Wight. “If you can imagine a fire happening in a zero-G environment, that’s an experiment that you just can’t do on Earth. But if you have a fire modeling code that’s robust enough, you can do a simulation where you just turn gravity off and see what happens. This kind of simulation technology will have the most impact in areas where you can’t do experiments.”
Modeling scenarios that can’t otherwise be explored is one of high-performance computing’s main advantages over physical experimentation. With the Uintah software, researchers now have a robust tool that will spur knowledge across many disciplines.
To learn more about this research, visit the C-SAFE homepage or explore their wiki to see more images and animations.
Aaron Dubrow
Texas Advanced Computing Center
Science and Technology Writer
April 15, 2009
The Ranger supercomputer is funded through the National Science Foundation (NSF) Office of Cyberinfrastructure “Path to Petascale” program. The system is a collaboration among the Texas Advanced Computing Center (TACC), The University of Texas at Austin’s Institute for Computational Engineering and Science (ICES), Sun Microsystems, Advanced Micro Devices, Arizona State University, and Cornell University. The Ranger and Lonestar supercomputers, and the Spur HPC visualization resource, are key systems of the NSF TeraGrid (www teragrid.org), a nationwide network of academic HPC centers, sponsored by the NSF Office of Cyberinfrastructure, which provides scientists and researchers access to large-scale computing, networking, data-analysis and visualization resources and expertise.
