Scalable and effective peer-to-peer desktop grid system

Abstract

We have designed a set of protocols that use peer-to-peer techniques to efficiently implement a distributed and decentralized desktop grid. Incoming jobs with different resource requirements are matched with system nodes through proximity in an N-dimensional Content-Addressable Network, where each resource type is represented as a distinct dimension. In this paper, we describe a comprehensive suite of techniques that cooperate to maximize throughput, and to ensure that load is balanced across all peers. We balance load induced by job executions through randomly generated virtual dimension values, which act to disaggregate clusters of nodes/jobs, and also by a job pushing mechanism based on an approximate global view of the system. We improve upon initial job assignments by using a job-stealing mechanism to overcome load imbalance caused by heterogeneity of nodes/jobs and stale load information. We also describe a set of optimizations that combine to reduce the system load created by the management of the underlying peer-to-peer system and the job-monitoring infrastructure. Unlike other systems, we can effectively support resource constraints of jobs during the course of load balancing since we simplify the problem of matchmaking through building a multi-dimensional resource space and mapping jobs and nodes to this space. We use extensive simulation results to show that the new techniques improve scalability, system throughput, and average response time.