Towards effective scheduling policies for many-task applications Practice and experience based on HTCaaS

Abstract

Inthispaper,weconductacomparativestudyofrelativelysimpleyeteffectiveschedulingpoli-ciesformany-taskapplicationswheremultipleuserswithvaryingnumbersoftasksareactivelysharingacommonsysteminfrastructure.Wehaveimplementedthreedifferentschedulingmech-anismsthatcanaddressfairnessanduserresponsetimerespectivelyinacommonmiddlewarestackcalledHTCaaS,whichisapilot-job–basedmulti-levelschedulingsystemrunningontop ofproduction-levelclusters.Asarepresentativecaseofourmany-taskapplications,wehaveleveragedthevirtualscreeningapplication,whichisacomputationaltechniqueusedindrugdis-coveryprocesstoselectthemostpromisingcandidatedrugsforinvitrotestingfrommillionsofchemicalcompounds,whichtypicallyrequiresasubstantialamountofcomputingresources andefficientprocessingofdockingsimulations.Ourcomparativeexperimentalresultsofdiffer-entschedulingpoliciesshowhowwecaneffectivelysupportmultipleusersinasharedresource environmentbybalancingbetweenusersatisfactionandoverallsystemperformance,provideguidelinestoimprovesystemutilization,andaddressadditionaltechnicalchallengestosupport variousmany-task applications.