Incorporating partitioning and parallel plans into the SCOPE optimizer

— Massive data analysis on large clusters presents new opportunities and challenges for query optimization. Data partitioning is crucial to performance in this environment. However, data repartitioning is a very expensive operation so minimizing the number of such operations can yield very significant performance improvements. A query optimizer for this environment must therefore be able to reason about data partitioning including its interaction with sorting and grouping. SCOPE is a SQL-like scripting language used at Microsoft for massive data analysis. A transformation-based optimizer is responsible for converting scripts into efficient execution plans for the Cosmos distributed computing platform. In this paper, we describe how reasoning about data partitioning is incorporated into the SCOPE optimizer. We show how relational operators affect partitioning, sorting and grouping properties and describe how the optimizer reasons about and exploits such properties to avoid unnecessa...

• Type: Conference
• Year: 2010
• Where: ICDE
• Authors: Jingren Zhou, Per-Åke Larson, Ronnie Chaiken
• Keywords: Data Partitioning, Database, ICDE 2010, Massive Data Analysis, SCOPE Optimizer