Optimizing data aggregation for cluster-based internet services

Large-scale cluster-based Internet services often host partitioned datasets to provide incremental scalability. The aggregation of results produced from multiple partitions is a fundamental building block for the delivery of these services. This paper presents the design and implementation of a programming primitive – Data Aggregation Call (DAC) – to exploit partition parallelism for clusterbased Internet services. A DAC request specifies a local processing operator and a global reduction operator, and it aggregates the local processing results from participating nodes through the global reduction operator. Applications may allow a DAC request to return partial aggregation results as a tradeoff between quality and availability. Our architecture design aims at improving interactive responses with sustained throughput for typical cluster environments where platform heterogeneity and software/hardware failures are common. At the cluster level, our load-adaptive reduction tree constr...

• Type: Conference
• Year: 2003
• Where: PPOPP
• Authors: Lingkun Chu, Hong Tang, Tao Yang, Kai Shen
• Keywords: DAC Request, Distributed And Parallel Computing, Global Reduction Operator, Partition Parallelism, PPOPP 2003