DIY: How to Build a Supercomputer

Construction of Frontera

Published on August 8, 2019 by Jorge Salazar


The construction of the Frontera supercomputer at the Texas Advanced Computing Center (TACC) involved multiple industry partners over the course of several months. Dell EMC delivered fully integrated racks. Mellanox provided the HDR InfiniBand network. Intel provided its 8280 Cascade Lake processors. DataDirect Networks supplied the storage appliances. Innovative cooling systems came from Cooltera and CoolIT to the racks; and Green Revolution Cooling with their immersion cooling system for the graphics processing units. And that's just to name a few.

A one-million-gallon thermal storage tank supplied by an onsite chilling station provides efficient cooling to Frontera, which helps keep its energy footprint small. Renewable energy from wind and solar meets a significant amount, about one-third, of Frontera's needs. In addition, TACC added infrastructure to handle Frontera's 5.9 megawatts of power, roughly equivalent to about 1,000 homes.

Other steps in readying the machine room for Frontera involved constructing stands to support its electrical transformers, which weigh thousands of pounds. And each rack required individual plumbing to be installed.

Once the plumbing and electrical infrastructure were ready, the first racks to arrive were the data storage servers. DataDirect Networks supplied a flash-based filesystem with their IME240 storage appliances, with four DDN ES18K EXAScaler appliances to be used for the home and scratch filesystems.

Next came the six Mellanox Quantum CS8500 core switches, capable of 320 terabytes per second switching capacity to the Mellanox HDR InfiniBand network to support all application communications between Frontera's compute nodes.

After the storage and network switches were in place, the first of 91 fully-loaded compute racks from Dell EMC arrived and were placed in the machine room. The compute racks consisted of a total of 8,008 dual-socket servers with Intel 8280 Cascade Lake processors, a proven technology that allows Frontera to reach a peak performance of 38.7 petaflops.

An experienced TACC staff installed over 50 miles of interconnect cable between the nodes and the network switches, aided by dozens of volunteers who helped unwind and deliver cables.

After round-the-clock work to stand the system up and test it, Frontera ran the LINPACK benchmark — number-crunching subroutines used by the international Top500 project to rank the fastest supercomputers in the world. NSF-funded Frontera earned a #5 spot on the Top500 list for June of 2019, giving The University of Texas at Austin two supercomputers (Stampede2, #19) ranked in the top 20 fastest in the world.

"Building Frontera was an incredible feat and required a big, expert team," said Dan Stanzione, Executive Director of TACC and Principal Investigator of the Frontera project. "Our ability to stand up such a complex machine under intense time pressure is something that sets TACC apart from other organizations in the world. We're proud to have executed and are now excited to see the amazing science it can enable."

Testing and early user access will continue through mid-summer of 2019 with full production expected for late summer 2019.