Hikari, an HPE Apollo 8000 system, is the result of a collaboration between TACC, the New Energy and Industrial Technology Development Organization (NEDO), a Japanese government agency, and NTT FACILITIES INC. Hikari should be the "greenest" of the TACC systems.
The Hikari system is part of a demonstration project to show off the potential of High Voltage Direct Current (HVDC) datacenters. Hikari will operate on a 380V DC distribution system. In addition to the system itself, we will be installing about 250KW of solar panels (over the parking lot of the new building -- enough to power the whole system), an HVDC UPS battery system, and HVDC CRAC (Computer Room Air Conditioner) units, as well as the power distribution equipment. Running all DC from the solar source through the UPS and directly to the racks saves four AC/DC conversions that would happen in a normal datacenter (the system will use grid power when solar is unavailable). The solar farm and HVDC equipment will then become a permanent part of TACC's data center infrastructure.
In addition to the HVDC components, the HPE Apollo 8000 has a number of other "green" features that should make this our most efficient system yet. The system continues the evolution of bringing water ever closer to the racks with self-contained water-cooled racks (that don't reject any heat into the datacenter air). Separate water loops move air directly across the processors – these loops are at near-vacuum pressures so the water will boil at low temperatures (around 105F or 40C) and convect to the sides of each blade, where it hits a heat exchanger, condenses, and is pushed back over the processors without the need for active pumps in each blade.
- 432 node HPE Apollo 8000
- Intel 12-core "Haswell" Xeon processors
- More than 10,000 cores
- Delivers a peak performance of over 400TF
- EDR Infiniband interconnect provided by Mellanox – TACC's first deployment of EDR
Hikari will be used to support TACC's secure/compliant computing needs (HIPAA/FISMA compliant data), as well as the increasing demand for fast turnaround on the many jobs received through web APIs from Science Gateways.