Comparing the performance of clusters, Hadoop, and Active Disks on microarray correlation computations

Abstract--Microarray-based comparative genomic hybridization (aCGH) offers an increasingly fine-grained method for detecting copy number variations in DNA. These copy number variations can directly influence the expression of the proteins that are encoded in the genes in question. A useful analysis of the data produced from these microarray experiments is pairwise correlation. However, the high resolution of today's microarray technology requires that supercomputing computation and storage resources be leveraged in order to perform this analysis. This application is an exemplar of the class of data intensive problems which require high-throughput I/O in order to be tractable. Although the performance of these types of applications on a cluster can be improved by parallelization, storage hardware and network limitations restrict the scalability of an I/O-bound application such as this. The Hadoop software framework is designed to enable data-intensive applications on cluster archit...

• Type: Journal
• Year: 2009
• Where: HIPC
• Authors: Jeffrey A. Delmerico, Nathanial A. Byrnes, Andrew E. Brunoh, Matthew D. Jones, Steven M. Gallo, Vipin Chaudhary
• Keywords: Active Disk, Cluster Architecture, Distributed And Parallel Computing, HIPC 2009, Number Variations