Researchers at The University of Texas at Austin's Center for Relativity (CfR) are using computers from the Texas Advanced Computing Center (TACC) to provide a better understanding of the interactions between spinning black holes.

The CfR research team, Director Richard Matzner, postdoctoral fellow Scott Hawley and undergraduate student Michael Vitalo, investigated the strength of the gravitational attraction between two black holes as the direction of each hole's spin changed, as a way of better understanding the dynamics of what are thought to be the strongest sources of gravitational waves detectable on earth. These findings are the subject of the authors' latest paper [1], and will ultimately expand our understanding of the universe.

The field of gravitational physics is in the midst of a great revival, largely driven by the construction of gravitational wave detectors such as LIGO (shown below).

These sophisticated laser interferometers are the most sensitive instruments ever designed by man (sensitive to 1 part in 10²¹) and are designed to measure tiny ripples in the fabric of spacetime, called gravitational waves, from distant astronomical sources. The strongest of these sources are binary black holes, which spiral in towards one another and merge to form a single black hole, all the while giving off strong gravitational waves.

To separate the astrophysical signals from background noise in the detector, scientists need to have a solid idea of what they're trying to find. This is a "needle in the haystack" problem of grand proportions, and sophisticated "template" waveforms are in great demand for use in picking out the true signals from the detectors' data.

These template wave forms can be obtained through closed-form analytical calculations when the holes are far apart and after the final merger occurs, but for the "in between" period, only numerical computations are able to provide the full solution to Einstein's nonlinear gravitational equations. Recently, the field of "numerical relativity" has itself been undergoing great change, as sophisticated 3-D simulations of binary black hole mergers have finally begun to simulate beyond a single orbit. Simulations of multiple orbits are needed for accurate waveforms; however, Einstein's nonlinear equations pose such significant challenges that up until the past year, all simulations would crash prior to a full orbit due to numerical instabilities.

Simulations performed by Frans Pretorius [2] using TACC's 1024-processor Linux cluster, Lonestar, represented the first full inspiral and merger simulation that lasted longer than an orbit. More recently, UT Brownsville [3] and NASA Goddard Space Flight Center [4] have extended these results using new "gauge conditions" for the coordinates of the simulation. These developments were featured in a recent science news article [5].

Almost all of these simulations, however, neglect the significant role that the black holes' spin will have on the evolution of the system. This current work by Hawley, Vitalo and Matzner aims to provide greater insight into the role of spin for binary black hole systems.

According to general relativity, the rotation of an object can have a gravitational effect on the object's surroundings, in addition to the usual gravitational attraction due to the object's mass. This latter effect is dominant, so two objects are always attracted, but in the case of two spinning masses, their spins can provide either an additional attraction or a small repulsion to the overall gravitational interaction. In this way, two spinning black holes can be loosely compared to a pair of magnets, which repel each other when their north poles are facing one another, and attract each other when opposite poles are near each other. The actual interactions for spinning black holes is more complicated than those of magnets, and the precise angular dependence of the spin-spin effects can only be computed either analytically in perturbative asymptotic regimes, or generally in full numerical simulations. The analytic expression for the spin-spin effect of two holes' spin on the "binding energy'' of the system was given by Robert Wald in 1972 [6].

The CfR research team looked at two different "series" of spin orientations. In both series, two black holes were placed a certain distance apart on the x-axis, each with a spin-per-unit-mass of 1/2 (it can range from -1 to 1, in normalized units where both the speed of light and Newton's gravitational constant are in unity), aligned in the positive z direction. In the first series, the angular momentum vector of the hole on the negative z axis was rotated in the x-z plane, i.e. tipping away from the other hole, as shown in the following figure. In the other series, the "yz series", the same hole was rotated in the y-z plane, i.e. in a plane perpendicular to the axis of separation between the holes.

The two different BBH configurations investigated. In all cases, the black hole on the positive x-axis is held fixed with a constant spin of in the z direction. In the "xz series," the spin axis of hole the hole on the negative x-axis rotated in the xz plane. In the "yz series," this spin axis is rotated in the yz plane.

The analytical formula of Wald predicted that both of these series would provide the same cosine curve on a graph of binding energy versus spin angle. The numerical calculations, however, revealed otherwise. The following graph shows Wald's cosine term as the black line for the yz series; the blue squares for the xz series show an additional sine-squared variation. Matzner was able to explain this as being due to the mass quadrupole moment of a spinning black hole; put in simpler terms, the shape of a spinning hole is not spherical, but rather oblate. This eccentricity in the shape of the hole affects the gravitational field around it just as the Earth's oblateness means that satellites don't orbit in a spherical potential. When you rotate the hole, the effect of this oblateness causes a variation in the gravitational field, which is expressed in the binding energy.

Graph of the "binding energy" (a measure of gravitational attraction) between two holes, as the spin axis of one hole was varied. The angle between the spin axis and the positive z-axis is plotted on the horizontal axis, and is measured in radians.

The overall strength of the gravitational attraction decreases (at large distances) as the reciprocal of the distance separating the holes, whereas these spin-spin affects are expected to decrease as the inverse cube of the separation. Unfortunately, the uniform-mesh ("unigrid") computer code had a hard time measuring the binding energy for large separations due to the multiple length scales significant in the problem. The binding energy is defined as the difference between something called the ADM Mass (a measure of the total matter-energy content of the spacetime) - which is ideally evaluated at spatial infinity but practically evaluated near the outer boundary of as large a computational domain as possible - and the sum of the individual holes' "horizon masses," which are evaluated close to the holes.

Thus, even using 513³ grid points for high-resolution simulations on TACC's Lonestar system, the present study was unable to accurately confirm the separation-dependence of the spin-spin effects. A new mesh-refined version of the code is nearly ready for production, and should be able to treat these divergent length scales much more efficiently than the unigrid code, thereby studying the separation-dependence accurately. Solving Einstein's tensor equations requires storage for the sixteen grid functions, and the multigrid method adds eight more. Nearly all of these need to be defined at as high a resolution as possible to resolve the black holes and have the outer boundaries far apart. This study used grids of 513³ grid points, which implies about 32 gigabytes of RAM needed for storage. This particular problem maps well with the distributed-memory architecture of the Lonestar cluster, and initial runs were carried out utilizing 32 processors each. The original version of this research code was ready for production in September 2005 and was deployed on TACC's Lonestar cluster. Throughout the project, TACC provided technical support to the researchers by providing recommendations for using the system effectively and helping to debug application issues as they arose during the deployment phase.

Lonestar is now on its way to becoming one of the most powerful supercomputers in the world. The Dell Linux cluster, which is being upgraded to 1,300 Dell PowerEdge 1955 blade servers, will provide a theoretical peak performance of more than 55 teraflops once the system achieves full production status on October 1. The architecture of the latest Intel Xeon dual-core processors and increased floating point and memory performance, combined with a high-speed Infiniband interconnect, will result in greater performance and scalability of applications that run on Lonestar.

"The upgrade to Dell PowerEdge processors will prove almost a factor 10 in throughput," Richard Matzner said. "This will mean the ability to simulate a much broader range of spin-spin configurations, and will allow much higher resolution and much higher precision simulations for the specific data that suggest the most interesting Physics."

The multi-resolution version of the code (pictured below) is nearly ready for production. This should allow researchers to resolve the black holes better, place the outer boundaries farther out (and the black holes farther apart), all using a fraction of the current memory requirements. This will translate into fewer processors needed, shorter queue times and more scientific throughput.

This initial code will then be interfaced with an evolution code for the full numerical simulation of binary black hole inspiral and merger. This evolution code is being developed by other members of Matzner's group, postdoc Andrea Nerozzi, and graduate student Paul Walter. This evolution effort is another significant user of TACC's resources and should enter production soon.

This research was supported through TACC allocations A-phaz and TG-PHY050037T, NSF grant PHY0354842, and by NASA grant NNG04GL37G. Portions of this work were conducted at the Laboratory for High Energy Astrophysics, NASA/Goddard Space Flight Center, Greenbelt Maryland, with support from the University Space Research Association. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing computational and storage resources that have contributed to the research results reported within this paper: http://www.tacc.utexas.edu

1. S.H. Hawley, M.J. Vitalo and R.A. Matzner, "Spin Dependence in Computational Black Hole Data", Submitted to Phys. Rev. D, available at gr-qc/0604100 (2006).
2. F.Pretorius, "Evolution of Binary Black Hole Spacetimes," Phys. Rev. Lett. 95 121101 (2005).
3. M. Campanelli, C. O. Lousto, P. Marronetti, Y. Zlochower, "Accurate Evolutions of Orbiting Black-Hole Binaries Without Excision", Phys. Rev. Lett. 96 111101 (2006).
4. J.G. Baker, J. Centrella, D.-I. Choi, M. Koppitz, J. van Meter, "Gravitational wave extraction from an inspiraling configuration of merging black holes," Phys. Rev. Lett. 96 111102 (2006).
5. R. Cowen, "Crash: Ripples of space-time debut in black holes simulations," Science News 169 (2006). Available (subscribers only) here.
6. R. Wald, "Gravitational Spin Interaction," Phys. Rev. D6 406 (1972).