From Katrina Forward: Producing Better Storm Surge Forecasts

ADCIRC: The Next Generation Storm Surge Model

Clint Dawson and colleagues are using Lonestar, a supercomputer at the Texas Advanced Computing Center, to develop a next-generation storm-surge model—one they hope replaces NOAA’s SLOSH model, the model currently used by hurricane forecasters.

The Atlantic hurricane season officially began on June 1 and the forecast from the NOAA National Hurricane Center calls for a very high likelihood (75% chance) of an above-normal hurricane season in 2007, with 13 to 17 named storms, seven to 10 hurricanes, and three to five major hurricanes. This outlook signifies a sharp increase in activity from the near-normal season observed in 2006. The question on everyone’s mind is—will the 2007 hurricane season be a repeat of the record-breaking 2005 season, one of the deadliest and costliest hurricane seasons in U.S. history?

Storm surge is a large dome of water, often 50 to 100 miles wide, that sweeps across the coastline where a hurricane makes landfall. The storm tide is the combination of the storm surge and the astronomical tide. Click image for more details and a larger view.

If Clint Dawson at The University of Texas at Austin has anything to do with it, the loss of life due to storm surge will be minimal in future hurricane seasons, even if there is another record-breaking season. Dawson and his collaborators have been hard at work improving the accuracy of their model, called the ADvanced CIRCulation Model for Coastal Ocean Hydrodynamics, or ADCIRC, which is used to predict storm surge height and velocity.

Storm surge is a large dome of water, often 50 to 100 miles wide, that sweeps across the coastline near where a hurricane makes landfall (National Weather Service, 2001). This surge of water, topped by battering waves and combined with high winds, is often deadly. It can cause great property damage, and often loss of life, as it moves ashore. Historically, storm surge has caused nine out of 10 hurricane fatalities (FEMA). Storm surge heights depend on the complex interaction of several variables, including the size of the storm, the central minimum pressure, the storm’s forward speed, the bathymetry (i.e. topography of the ocean bottom, especially near the point of landfall), the timing of the astronomical tides, and most importantly, the maximum wind speed of the storm.

“We started using Lonestar in November 2006—soon after it came on line—and we have used it 'pedal to the metal' ever since.” –Clint Dawson.

The Evolution of ADCIRC

Originally developed by Joannes Westerink, a civil engineer at the University of Notre Dame, and Richard Luettich, a marine scientist at the University of North Carolina, ADCIRC is a two-dimensional numerical model that predicts long-term periods of circulation along coastal shelves, coasts, and estuaries. The starting point for ADCIRC is the “shallow water equations,” a set of partial differential equations that solve for water elevation and velocity, which can be used to predict storm surge.

ADCIRC can be modified for a number of applications—essentially any phenomena related to the hydrodynamics of surface waters such as an ocean, river, or lake. For hurricane storm surge and flooding predictions, inputs to the model include such variables as coastline geometry, water depths, land elevations and obstructions, land types, and hurricane winds and pressures.

Dawson first collaborated with Westerink and Luettich on ADCIRC in the mid-1990s, when they undertook a joint project to parallelize the code. At the time, Dawson was in the Department of Mathematical Sciences at Rice University and a member of the Center for Research on Parallel Computation. Dawson started with the existing serial code and distributed the work among processors, each with its own memory. “The code that we developed is highly scalable, up to a certain number of processors, depending on the size of the problem that we are solving,” said Dawson. He has been a member of the ADCIRC development group ever since.

Clint Dawson is a Professor in the Department of Aerospace Engineering and Engineering Mechanics at The University of Texas at Austin. He is affiliated with the Center for Subsurface Modeling in the Institute of Computational Engineering and Sciences. Image courtesy of Jennie Trower.

Once they had the parallel code, the ADCIRC development group realized they could do much finer grid simulations then they had ever done before, which would greatly improve the accuracy of the model. “In the mid-1990s, our biggest runs were perhaps 20,000 nodes (or 60,000 degrees of freedom) on a workstation, but it would take three days to run it. Once the code was parallelized, we could go up to several hundred thousand degrees of freedom, and it would run in just a few hours. It was a huge leap in computational ability,” said Dawson.

Simulating Storm Surge for Southern Louisiana

In the late 1990s, the U.S. Army Corp of Engineers New Orleans District asked the ADCIRC development group to develop a storm surge model for New Orleans and the adjoining southern Louisiana coast. The Corps was studying the southern Louisiana levee system and wanted to know how it would fare if it took a direct hit from a major hurricane. Since then, the group has constructed a sequence of models for southern Louisiana with varying degrees of detail and resolution, which they continue to refine today.

One clear advantage of using ADCIRC to model the Louisiana coastline is that it is capable of simulating storm surge predictions over a very large computational domain, and at the same time, generating high-resolution output in coastal areas with complex shorelines and bathymetry.

“There will be interest in Katrina and what happened in southern Louisiana, Mississippi, and Alabama for many years to come. In fact, the real work to understand what happened and what to do next has really just begun,” said Clint Dawson.

For the southern Louisiana model, the computational domain incorporates the entire Gulf of Mexico, the Caribbean Sea, and a large portion of the western North Atlantic Ocean. By using a much larger domain than previous storm surge models, ADCIRC more accurately simulates the storm surge as it propagates from deep waters in the open sea onto the shallower continental shelf and adjacent coastal floodplain.

The computational domain or boundary is divided into a grid of nodes or “computational points.” By 2005, the model had 314,442 nodes, containing environmental data such as water depth, land structures, topography, and other variables that would affect the storm’s profile.

The computational domain (with bathymetry) of ADCIRC includes the Gulf of Mexico, the Caribbean Sea, and a large portion of the western Atlantic. Image courtesy of Clint Dawson. Click image for larger view.

ADCIRC utilizes an unstructured (finite-element) grid where the size of the mesh ranges from kilometers in the deep ocean to 50-100 meters in some of the coastal areas around New Orleans. In other words, the nodes contain much more refined and detailed information as they approach the shoreline. The boundary around New Orleans extends all the way to Baton Rouge, and includes information on topography, land use, canals, levees, raised highways, and other land characteristics that might affect storm surge.

“The storm surge model involves solving for the circulation, but then we couple that to the storm track and wind model that we get from the National Weather Service to generate a wind field, and it serves as a forcing term for our model,” said Dawson.

In addition to storm track and wind data, any feature that will obstruct the flow of water must be included in the model—a raised roadway, for example. “But such obstructions are at such a fine scale—a small, narrow strip of land on a huge landscape—that it is a challenge to account for that in our model,” said Dawson.

TACC’s Role

The unstructured, finite-element grid used by ADCIRC laid over for the entire computational domain. Note that nodes in the open ocean have a much lower resolution than those close to shore. Red lines represent raised features such as levees or raised roads that might slow or stop hurricane storm surge. Image courtesy of Clint Dawson. Click image larger view.

To refine this complex model, Dawson and his collaborators are using the high-performance compute and data resources at the Texas Advanced Computing Center (TACC) at The University of Texas at Austin. Since October 2006, Dawson’s research group has run approximately 4,400 cases on TACC’s Dell HPC cluster, Lonestar, one of the most powerful computing systems for open academic research in the world. Each case required up to 256 cores of the 5,840-core cluster. With advice from TACC’s High Performance Computing group, Dawson improved the performance of ADCIRC by using Lonestar more efficiently, and now runs ADCIRC simulations much faster than ever before. To date, Dawson’s project has consumed more than 1.1 million processor hours on Lonestar.

“We started using Lonestar for this project in November 2006—soon after it came on line—and we have used it 'pedal to the metal' ever since,” said Dawson. “Typically, we run on 256 processors because we see a drop-off in performance when going from 256 to 512 processors. However, this summer, we are doing some performance studies and attempting to optimize the code to scale to 1,024 processors. We hope to have some results soon.”

Refining the Model: Hindcasting Katrina

In 2004, FEMA, along with other federal and State of Louisiana entities, used ADCIRC to model how New Orleans might flood if a major hurricane, the “Big One” were to strike (FEMA, 2004). They developed a hypothetical hurricane named Pam—modeled as a slow moving, Category 3 storm.

On August 29, 2005, Hurricane Katrina made landfall as a strong Category 3 storm in southeastern Louisiana. Having been a large, Category 5 storm several days earlier, Katrina produced record storm surges along the Mississippi coast. Image courtesy of NOAA. Click image for larger view.

This simulation was devastating, and eerily prophetic, because it showed that the levees in the New Orleans area were simply too low to keep back the storm surge from a hurricane of even moderate strength. The model predicted that if a Category 3 storm or stronger approached New Orleans from the southeast, the storm surge would likely push into Lake Pontchartrain and up the Mississippi River. The levees around the lake and river would be overwhelmed and water would enter the city, filling it up like a bowl. The situation was even gloomier than the simulation showed because Pam was based on inaccurate elevations due to continual land subsidence; the ground is lower in many places by as much as two feet.

“When Katrina happened in 2005, there was suddenly lots of publicity about Hurricane Pam. Pam clearly showed that there was an impending catastrophe in New Orleans—so, of course, journalists and others were asking, why wasn’t something done then? Everyone, including the Army Corps of Engineers, the City of New Orleans, and the State of Louisiana, had egg on their face because this model was out there and they didn’t pay attention to the results of the Hurricane Pam study,” said Dawson.

Although the flooding in New Orleans received much of the media attention, direct damage from Katrina’s storm surge was most devastating in other parts of Louisiana, and along the coast of the neighboring states of Mississippi and Alabama. Storm surge flooding of at least 25 to 28 feet above the normal tide level occurred along portions of the Mississippi coast, and 10 to 20 feet above normal tide levels along the southeastern Louisiana coast. In Bay St. Louis, Mississippi, Hurricane Katrina generated the largest recorded storm surge in the U.S. history at more than 34 feet.

The damage in Long Beach, Mississippi, near Gulfport, as seen on September 6, 2005, caused by the estimated 25 to 35 foot storm surge produced by Hurricane Katrina. Click image for more details and a larger view.

The storm surge damage extended several miles inland along the entire Mississippi coast, causing the total destruction of all nearly all structures in its path. There were 200 deaths in Mississippi contributed to Katrina, and almost all of these drowned in the storm surge. Similar damage and deaths occurred in rural coastal areas of southeastern Louisiana.

The Army Corps of Engineers turned to the ADCIRC development group to help them understand what went wrong during Katrina.

Originally, ADCIRC focused only on southern Louisiana, but to model Katrina, Dawson and his colleagues had to extend it to include southern Mississippi and Alabama. In addition, the resolution of the model needed to be increased, so the levee systems and other land features could be better resolved. As a result, the model became much more complicated—with many additional features compared to earlier versions of the model.

Peak storm surge elevation, as simulated by ADCIRC, during the Hurricane Katrina event (shown in meters). To see an animated version of the hindcast, go to http://www.nd.edu/~adcirc/katrina.htm. Image courtesy of Clint Dawson, and Joannes Westerink, University of Notre Dame. Click image for larger view.

Today, ADCIRC is being used in southern Louisiana by the U.S. Army Corps of Engineers to design levee heights and alignments, by FEMA to establish flooding probabilities for insurance purposes, by the State of Louisiana to predict hurricane inundation, and by the Louisiana State DNR to assess coastal restoration projects.

Model Refinements: Making the Leap to Forecasting

Currently, NOAA’s National Weather Service, local forecasting offices, and other agencies across the country concerned with emergency preparedness use NOAA’s SLOSH (Sea, Lake, and Overland Surges from Hurricanes) model to forecast storm surge heights. “For forecasting, SLOSH is still widely used—primarily because it is fast—but, it is also readily available and everybody already knows how to use it,” explained Dawson.

Once the National Weather Service generates a new storm track, it takes only 20 to 30 minutes to run the data through SLOSH to forecast the storm surge along a section of coastline. ”The problem,” says Dawson, “is that the model is also very coarse—the elements and grid used in the model are huge, so it only gives a very rough, back-of-the envelope estimate.”

However, except for the Louisiana coast, SLOSH remains the only operational model used for forecasting storm surges. Every time the Weather Service updates a hurricane’s storm track, which gets more frequent as the storm approaches landfall, SLOSH is used to update the storm surge forecast. “Currently, our model just cannot match its turnaround time,” said Dawson, “but we are working hard to improve it.”

The image shows the details of the unstructured grid used in ADCIRC in southeastern Louisiana. The red lines represent raised features such as levees and raised roads, which figure into the storm surge model. Also, notice that the grid elements in the open ocean (right, bottom corner) are much larger then those near the coast or on land. Image courtesy of Clint Dawson. Click image for larger view.

The main sticking point is visualizing the output of the model. “Currently, our model runs in 56 minutes, but we generate huge amounts of data that must be processed and visualized, which currently takes several hours. We need to come up with a better way of quickly getting the data off the machine, getting it on a machine where we can visualize it, and then effectively communicating the forecast to the emergency management people—wherever they might be. We need fast, real-time turnaround of information. That’s a huge issue for us—and I can’t stress that enough,” exclaimed Dawson. Dawson hopes to work with TACC scientific visualization staff in the near future to find ways to visualize their data more quickly.

In the future, Dawson would like to see ADCIRC become the model of choice for forecasters. According to Dawson, “as we go to petascale computing, we should be able to obtain a 20 minutes turnaround with our model and with the level of detail that would really make a difference to emergency managers and to first responders—that’s my goal—whether it takes 10,000 processors or 50,000 processors—so what? If we can get turnaround in 20 minutes, at much finer scales than SLOSH, it would be an amazing improvement over how storm surge is forecast now.”

"We can’t wait until TACC’s Ranger comes on line—we are chomping at the bit! I’m not kidding!" –Clint Dawson

“Ranger” is TACC’s new supercomputer system specifically designed to support very large science and engineering computing requirements like those of Dawson. “We can’t wait until TACC’s Ranger comes on line—we are chomping at the bit! I’m not kidding,” said Dawson. When it launches in late 2007, Ranger is expected to be the world’s most powerful supercomputer with more than 500 teraflops peak performance (a teraflop is a measure of a computer's speed and can be expressed as one trillion calculations per second). Ranger will enable breakthrough science that has never before been possible. To deploy and support Ranger, the National Science Foundation awarded TACC $59 million, the largest award ever from that agency to UT Austin.

However, Dawson believes that ADCIRC still has a way to go before it can be widely used in forecast mode. According to Dawson, “ADCIRC is certainly a candidate model, but we first need to create grids for the East Coast and for other coastlines susceptible to hurricanes. We need to make our model more of a “black box” so it can be used for other regions and coasts. It’s not quite that black box yet.”

The Future: The Next Generation Storm Surge Model

Dawson and the ADCIRC development group are currently working on several fronts to refine ADCIRC further. Their long-term goal is to continue to improve the code by putting in better or more robust algorithms, adding more fine-scale features and physics, and exploiting high performance computing as much as possible for very high-resolution studies. “There is a lot to be learned from these studies that we haven’t even thought about yet. Every time we add more details to the model, we learn more about what’s missing and what needs to be done,” said Dawson.

Clint Dawson (middle) and Joannes Westerink (right) speak with Congressman Vernon Ehlers (R-MI) in November 2005 at the American Mathematical Society’s briefing on Capital Hill. Dawson and Westerink, members of the ADCIRC development group, spoke to Congressional representatives about how mathematics helps predict storm surges. “At the time,” said Dawson, “the nation was still embroiled in Katrina and the events surrounding it, so the room was absolutely overflowing.” Image courtesy of the American Mathematical Society.

The next iteration of the model partially uses LIDAR, a remote sensing system that collects high-resolution topographic data using lasers, to establish topography, and broad levee and road heights. It has grid resolution to 60 m and defines 2.1 million computational nodes, nearly an order of magnitude better than the 2005 model. The ADCIRC development group is working on a wider Gulf Coast model intended to provide high-resolution coverage along the Gulf Coast from Brownsville, Texas to Mobile Bay, Alabama.

“Having grown up in Texas, I would like to see us develop as fine-scaled, detailed, and robust model for Texas as they now have for Louisiana,” said Dawson.

Dawson’s personal goal is to refine the model such that forecasts can be made for individual neighborhoods, subdivisions, and streets. “If we can do that, officials would not have to evacuate an entire city, as they did in Houston when Hurricane Rita was approaching landfall. In the future, evacuations will be much more pinpointed.”

by Wendy E. Sera
Science & Technology Writer
Texas Advanced Computing Center
July 2007

TACC User Profile:
Dr. Clint N. Dawson

Dr. Dawson is a Professor in the Department of Aerospace Engineering and Engineering Mechanics at The University of Texas at Austin (UT Austin), and is affiliated with the Center for Subsurface Modeling in the Institute of Computational Engineering and Sciences.

Dawson received his B.A. (summa cum laude) in mathematics from Texas Tech University in 1982 and his M.S. in mathematics from Texas Tech University in 1984. He received his Ph.D. in mathematical sciences from Rice University in 1988. From 1988-90, he was a National Science Foundation Postdoctoral Fellow and Dickson Instructor in the Department of Mathematics at the University of Chicago. In 1990, he returned to Rice as an assistant professor in the Department of Computational and Applied Mathematics. He was promoted to associate professor in 1994 and in 1995 moved to the College of Engineering at The University of Texas at Austin, where he holds the Temple Foundation Endowed Faculty Fellowship No. 7. He was promoted to full professor in 2000.

Currently, he teaches a sophomore-level course in engineering dynamics and a graduate-level course in applied mathematics. He is also a graduate advisor in the Computational and Applied Mathematics Program at UT Austin.

Dawson has authored or co-authored over 100 technical articles in the areas of numerical analysis, numerical methods, environmental modeling, and large-scale parallel computing, with applications to flow and transport in porous media, and shallow water systems. In addition, he has been an author/co-author or editor/co-editor on eight book chapters, seven refereed conference proceedings, and 33 other major publications.

In 2001, he was elected Chair for the Society for Industrial and Applied Mathematics Activity Group on Geosciences, and has served on numerous conference organizing committees and review panels. Currently, he is on the editorial boards for the journal Advances in Water Resources and the SIAM Book Series Computational Science and Engineering. He is a Co-Managing Editor for the journal Computational Geosciences. He has been a reviewer for over 21 professional journals, and for granting agencies such as the National Science Foundation and the Department of Energy.

Selected Publications

Aizinger, V., and Dawson, C., “The Local Discontinuous Galerkin Method for Three Dimensional Shallow Water Flow,” Comp. Meth. Appl. Mech. Eng., 196, pp. 734-746, 2007.

Dawson, C.N., Editor, Special Issue on Discontinuous Galerkin Methods, Comp. Meth. Appl. Mech. Eng., Vol. 195, 2006.

Dawson, C. and Proft, J., “Discontinuous and coupled continuous/discontinuous Galerkin methods for the shallow water equations,” Comp. Meth. Appli. Mech. Eng., Vol. 191, pp. 4721-4746, 2002.

Dawson, C. and Proft J., “Discontinuous/continuous Galerkin methods for coupling the primitive and wave continuity equations of shallow water,” Comp. Meth. Appl. Mech. Engr., Vol. 192, pp. 5123-5145, 2003.

Dawson, C., Westerink J., Pothina, D. and Feyen, J., “Continuous, Discontinuous, and Coupled Discontinuous-Continuous Galerkin Finite Element Methods for the Shallow Water Equations,” Int. J. Num. Meth. Fluids, 52, pp. 63-88, 2006.

Kubatko, E., Westerink, J., and Dawson, C., “An Unstructured Grid Morphodynamic Model with a Discontinuous Galerkin Method for Bed Evolution,” Ocean Modeling, Vol. 15, pp. 71-89, 2006.

Westerink, J. J., Feyen, J. C., Atkinson, J. H., Luettich, R. A., Dawson, C. N., Powell, M. D., Dunion, J. P., Roberts, H. J., Kubatko, E. J., and Pourtaheri, H., “A New Generation Hurricane Storm Surge Model for Southern Louisiana,” Bulletin of the American Meteorological Society, in review

Wheeler, M. F., Lee, W., Dawson, C. N., Arnold, D. C., Kurc, T., and Parashar, M., “Parallel computing in environment and energy,” Sourcebook of Parallel Computing, J. Dongarra, I. Foster, G. Fox, W. Gropp, K. Kennedy, L. Torczon, and A. White eds. Morgan Kaufman Pubs., Boston. Chapter 6, pp. 145-166, 2003.