Latest News


Texascale Days Showcases Potential of Frontera for Large-scale Simulation and Modeling

Published on April 30, 2020 by Aaron Dubrow

A beginning stage in the transient readjustment of the distribution of mixed, relatively buoyant material from a model 25 solar mass (Mʘ) star. [Credit: Paul Woodward, University of Minnesota]

Computational simulation and modeling is always pushing the boundaries of the possible, adding more physical realism — down to the quantum level — or modeling larger systems — whether all the stars in a galaxy or all the interacting atoms in a cell.

As one of the largest supercomputers on the planet, Frontera allows researchers to solve problems that have been beyond the grasp of researchers in the past.

From March 10-16, 2020, the Texas Advanced Computing Center (TACC) hosted its second Texascale Days event, where research teams were provided exclusive access to Frontera, which is typically shared among dozens of projects.

"The event allows researchers to spread their wings, test the potential of their codes, and solve bigger problems than ever before," said John Cazes, director of High Performance Computing at TACC.

Seven teams used large portions of Frontera, or even the entire machine, to compute at a scale previously impossible. Here's a snapshot of what they achieved.

Evolving Stars

For Paul Woodward, the Texascale Days runs represented an opportunity to explore massive stars near the ends of their lives, and to test and validate 1-D convective boundary mixing — how the layers and elements interact — in massive stars. 3-D models can be too computationally expensive to model with full physics, so if a 1-D model can capture the dynamics of a system accurately, it would go a long way toward answering a number of astrophysics problems.

Asteroseismology, or the study of oscillations in stars, is one of very few ways that scientists can observationally test models of convective boundary mixing. Stars can be observed by the TESS (Transiting Exoplanet Survey Satellite) satellite using asteroseismology, which means models can be validated with confidence.

"The effectiveness of the mixing and the associated thickness of the mixing layer near the convective boundary produces, over the life of the star on the main sequence, a signature in the spectrum of wave modes that the satellite observes at the star's surface as a part of its search for planets orbiting other stars," Woodward explained. "Otherwise, we rely on comparisons of the 1-D models to detailed 3-D simulations like the ones we are performing on Frontera at TACC."

Woodward's team tested their code on up to 7,500 nodes on Frontera (out of 8,008), and then ran multiple jobs using an improved model with extra physics on 986 nodes. They computed for 60 straight hours over the weekend of the Texascale Days period.

"The animations we developed help to indicate where, in the 3-D space, the visible features are," Woodward wrote. "The vorticity images clearly show the fully developed turbulence of the convection zone. We are processing the half petabyte of data that our Texascale simulations produced, but in these images and the accompanying movie, one can get a first glimpse at the richness of this unique data set."

Cellular Gatekeepers

Aleksei Aksimentiev and his team at the University of Illinois, Urbana-Champaign used their Texascale Days' time to model the nuclear pore complex, which serves as a passageway into and out of the nucleus of a cell.

An image from the Aksimentiev group's simulation of the nuclear pore complex surrounded by a dense, cytoplasm-like environment, created from their Texascale Days run. [Credit: Aleksei Aksimentiev, UIUC]

"Scientifically, we are investigating how cytoplasmic proteins interact with the nuclear pore complex's central mesh, paying particular attention to a class of proteins called transport factors, which are known to ferry other cargo across the central channel," said David Winogradoff, a post-doctoral research fellow in Aksimentiev's group.

Referring to the visualization (right) that they produced, he said: "We've modeled this protein assembly, which contains a structured component [cyan surface] and a central disordered mesh [purple surface], along with cytoplasmic proteins throughout the system [many colors], mimicking the dense environment found inside a living cell."

Their simulations accounted for every atom of the system — 150 million atoms in total — including the nuclear envelope and surrounding water and ions. They used 4,000 nodes, half of the Frontera supercomputer, and saw good scaling, or efficiency, as they moved from smaller to very large scale computations.

Hypersonic Aircraft

Turbulent flow modeling is a common problem that supercomputers are used for. Daniel Bodony's team, also at University of Illinois, Urbana-Champaign, used Frontera to simulate flow conditions and model geometry in order to match, and compare results to, an experiment being conducted in the NASA Langley 20-inch Mach 6 tunnel. Bodony, the Blue Waters Associate Professor of Aerospace Engineering, studies fluid-thermal-structure interactions, one of the principal challenges that inhibits hypersonic vehicle design.

Their simulations used up to 3,982 nodes and explored the impact of a Mach 6 flow on a 35 degree ramp using direct numerical simulation. The geometry of the ramp represents a deflected control surface on a hypersonic vehicle, the component on an airplane that allows the pilot to adjust and control the aircraft's flight attitude.

"Our allocated time on Frontera will enable us to understand how hypersonic vehicles interact with the very fast, hot, and turbulent flows they generate," Bodony said. "We are especially interested in predicting and modeling how the vehicle's external surface responds — deforms and heats-up — to the high-speed flow, as well as how the surface changes impact the flow itself."

The Texascale Days' simulations used a rigid ramp; Bodony and his team will continue their work in the coming year using Frontera to model a flexible ramp that more closely matches reality.

Massive Coronavirus Simulations

Read how Texascale Days helped kick-start Rommie Amaro's efforts to create an all-atom model of the SARS-CoV-2 virus.

Modeling the Evolution of Galaxies

A powerful new generation of ground and space-based observatories will soon begin exploring the earliest stages of the formation of galaxies and quasars in the universe. These observatories, including the Large Synoptic Survey Telescope and the James Webb Space Telescope, will revolutionize our understanding of the origin of galaxies and quasars, and help constrain the nature of the dark matter, which is dominant in the universe.

Image of an Enzo-E cosmological adaptive mesh refinement (AMR) simulation on 3,800 nodes of Frontera performed during the Texascale Days event. The image shows the AMR mesh at multiple levels of enlargement superimposed on the base mesh of size 20483. Each little square represents the projection of a 163 mesh block. [Credit: Michael Norman, UCSD]

Detailed physical simulations that model the formation of these objects are an indispensable aid to understanding the coming glut of observational data, and to maximize the scientific return of these instruments.

Michael Norman, director of the San Diego Supercomputer Center and an early Frontera user, dedicated his Texascale Days' time to testing the open-source Enzo adaptive mesh refinement hydrodynamic cosmology code.

Adaptive mesh refinement, or AMR, refers to the method by which large volumes of space are divided into smaller parts that can be computed independently and then integrated to get a holistic answer. Doing so efficiently and accurately is a major challenge in nearly all fields of science. Designed to scale to millions of processors, the new framework, called Enzo-E, targets exascale high performance computing systems of the future.

Using Frontera, Norman explored the assembly of the first generation of stars and black holes into the first galaxies. He ran successfully on 3,800 nodes and was able to perform a 20483 AMR cosmology simulation — among the largest ever attempted for this problem.

Enzo-E's first science application will be to simulate the formation of the first generation galaxies in large cosmological volumes. This will improve on simulations performed using the old Enzo code which were limited to small cosmological volumes because of the limited scalability of its AMR infrastructure (which only scaled up to a 2563 AMR mesh).

"The great leap forward with Enzo-E is the ability to perform huge adaptive mesh refinement simulations," Norman said. "The larger mesh will allow us to simulate 512 times the volume, which in turn allows us to model the formation of the brighter, more massive galaxies that will be imaged by the James Webb Space Telescope."

Unraveling the Complexity of the Most Common State of Fluid Motion in the Universe

The lack of understanding of turbulence is a well-known limiting factor in predicting and controlling numerous processes in nature and engineering. In many of these, which include aerodynamic drag in transportation systems, energy generation, and even astrophysical phenomena, fluids experience density changes, which makes problems even more difficult to solve.

Visualizations generated during Texascale Days highlighting the structure of turbulent flows at different conditions. (Colors are magnitude of pressure gradients.) From left to right, one can see more compression waves (shockwave-like structures which look like bright lines) gaining prominence in the dynamics of the flows. [Credit: Diego Donzis, Texas A&M University]

"Frontera is allowing us to study the details of fundamental processes in these flows at conditions not possible before," said Diego Donzis, an associate professor of Aerospace Engineering at Texas A&M University. "We are able, for the first time, to access regimes in terms of turbulence strength (high Reynolds numbers) and compressibility levels (high Mach numbers) which is helping us understand universal features shared by processes involved in a car engine and those involved in the creation of structure in the observable universe."

The research is especially critical given that controlled experiments are exceedingly challenging and thus virtually non-existent. Simulations on Frontera, like those during Texascale Days which utilized more than 4,600 nodes, are offering us the possibility of exploring, for the first time, true inertial-range behavior for both hydrodynamic and thermodynamic variables around which physically-realistic models can be finally constructed. Thus, in addition to further our understanding of turbulence, these simulations will provide invaluable data to create better turbulence models which will then translate into faster vehicles, more efficient energy generation, improved industrial processing, and a better understanding of our universe.

Understanding Extremes in Turbulence

P.K. Yeung, a professor of aerospace engineering at Georgia Tech, has a long-standing interest in studying intermittency in fluid turbulence at relatively high Reynolds numbers using high resolution numerical simulations based on exact equations.

Fluid elements in a turbulent flow are constantly subjected to strong local deformation by the turbulence, which can be manifested in extreme events that are highly localized but with random amplitudes reaching up to thousands of times the average. These extreme events are difficult to resolve and sample, especially at high Reynolds number, but have great importance for applications such as combustion instability and the preferential concentration of inertial particles in turbulence.

Texascale Days enabled large-scale simulations by Yeung's group, with excellent performance when running on more than 1,100 nodes. The work will help answer key questions pertaining to the fundamental nature of turbulent flows.

"Massive simulations facilitated by the size of Frontera are helping advance our understanding of extreme events in turbulent flows," Yeung said.

Benefits to TACC

In addition to enabling novel, large-scale science, TACC's Texascale Days events provide an opportunity to push Frontera's capabilities in terms of whole system communication, reliability, and scale. Doing so lets all Frontera users realize the full potential of the system.

"Events such as Texascale Days allow our colleagues to experiment beyond their normal research and push their applications and our systems to the limit," said Cazes. "These are valuable exercises for our research colleagues and our center because the issues we encounter and overcome provide us with insight on how to better deploy our systems and support their research. These events are stepping stones to the science problems of tomorrow and the HPC resources required to investigate them."

Story Highlights

From March 10-16, 2020, TACC hosted its second Texascale Days event.

Seven research teams were provided exclusive access to Frontera, and used large sections or even the entire system to solve problems in astronomy, biology, and physics.

The event is a stepping stone to the science problems of tomorrow and the HPC resources required to investigate them.

It also provide TACC staff with insights into how to better deploy HPC systems and support research.


Faith Singer-Villalobos

Communications Manager | 512-232-5771

Aaron Dubrow

Science And Technology Writer

Jorge Salazar

Technical Writer/Editor | 512-475-9411