Karnak

Purpose

The Karnak project investigates how machine learning techniques can be use to optimize the use of distributed infrastructures. The project first focuses on developing and evaluating techniques to predict properties such as queue wait times, application execution times, and data transfer times in a distributed system. Second, the project analyzes how to apply such predictions so that computational scientists can efficiently use such infrastructures.

A major component of the Karnak project is the XSEDE Karnak service, which provides predictions of when the batch schedulers that manage XSEDE resources will start jobs. This service provides predictions for jobs before they are submitted (to help a user decide where to submit) and after they are submitted (to help users plan their work around when their analyses are expected to start).

Funding Source(s)

  • ECSS, SD&I

Related Link(s)

Publications

Ye Fan, Sudhakar Pamidighantam, Warren Smith. Incorporating Job Predictions into the SEAGrid Science Gateway. In Proceedings of the XSEDE'14 Conference, July 2014.

W. Smith. A Service for Queue Prediction and Job Statistics. In Proceedings of the 6th Gateway Computing Environments Workshop (GCE'10). November 2010.

W. Smith. The Karnak Prediction Service. In Proceedings of the TeraGrid'10 Conference. August, 2010. (extended abstract)

W. Smith. Prediction Services for Grid Computing. In Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium. April 2007.

Tim Cockerill

Director Of Center Programs
cockerill@tacc.utexas.edu | 512-553-0363