A Flexible Open-Source Toolbox for Scalable Complex Graph Analysis

The Knowledge Discovery Toolbox (KDT) enables domain experts to perform complex analyses of huge datasets on supercomputers using a high-level language without grappling with the difficulties of writing parallel code, calling parallel libraries, or becoming a graph expert. KDT provides a flexible Python interface to a small set of high-level graph operations; composing a few of these operations is often sufficient for a specific analysis. Scalability and performance are delivered by linking to a state-of-the-art back-end compute engine that scales from laptops to large HPC clusters. KDT delivers very competitive performance from a generalpurpose, reusable library for graphs on the order of 10 billion edges and greater. We demonstrate speedup of 1 and 2 orders of magnitude over PBGL and Pegasus, respectively, on some tasks. Examples from simple use cases and key graphanalytic benchmarks illustrate the productivity and perforalized by KDT users. Semantic graph abstractions provide bo...

• Type: Journal
• Year: 2012
• Where: SDM
• Authors: Adam Lugowski, David M. Alber, Aydin Buluç, John R. Gilbert, Steve Reinhardt, Yun Teng, Andrew Waranis
• Keywords: Data Mining, Graph Algorithm, Graph Operations, Open Source Code, SDM 2012