Attaching Cloud Storage to a Campus Grid Using Parrot, Chirp, and Hadoop

The Hadoop filesystem is a large scale distributed filesystem used to manage and quickly process extremely large data sets. We want to utilize Hadoop to assist with dataintensive workloads in a distributed campus grid environment. Unfortunately, the Hadoop filesystem is not designed to work in such an environment easily or securely. We present a solution that bridges the Chirp distributed filesystem to Hadoop for simple access to large data sets. Chirp layers on top of Hadoop many grid computing desirables including simple deployment without special privileges, easy access via Parrot, and strong and flexible security Access Control Lists (ACL). We discuss the challenges involved in using Hadoop on a campus grid and evaluate the performance of the combined systems.

• Type: Journal
• Year: 2010
• Where: CLOUDCOM
• Authors: Patrick Donnelly, Peter Bui, Douglas Thain
• Keywords: CLOUDCOM 2010, Distributed And Parallel Computing, Hadoop, Hadoop Filesystem, Large Data Sets