A Simulation-Provenance Data Management System for Efficient Job Execution on an Online Computational Science Engineering Platform

Abstract

In the past few years an online simulation service platform (named EDISON) has been applauded by several computational science and engineering communities in several countries. Though armed with multiple computing clusters and high-end storage resources, the platform has suffered from handling a huge amount of CPU-/IO-bound simulations that are most duplicated. Such intense simulations are normally admitted with no duplicate elimination and thus can adversely affect the performance of the platform. To address this performance concern, we propose a novel system, termed SUPERMAN, to seamlessly record and retrieve the provenances of previously executed simulations, and so prevent users from initiating duplicate and/or similar simulations using the limited computing resources. The system collects the simulation provenances based on a variant of a de-facto standard form, thereby offering interoperability. Based on the stored provenances, the system can provide useful simulation run statistics for users that need assistance. SUPERMAN also applies a hash-based duplicate elimination technique, resulting in making more efficient simulations on the platform. Finally, we show that the proposed proposed system could remove slightly over half of duplicate simulations on a variety of simulation software while obtaining about overall elapsed time savings of 30% and queuing time savings of 25%.