Run-time operator state spilling for memory intensive long-running queries

Main memory is a critical resource when processing longrunning queries over data streams with state intensive operators. In this work, we investigate state spill strategies that handle run-time memory shortage when processing such complex queries by selectively pushing operator states into disks. Unlike previous solutions which all focus on one single operator only, we instead target queries with multiple state intensive operators. We observe an interdependency among multiple operators in the query plan when spilling operator states. We illustrate that existing strategies, which do not take account of this interdependency, become largely ineffective in this query context. Clearly, a consolidated plan level spill strategy must be devised to address this problem. Several data spill strategies are proposed in this paper to maximize the run-time query throughput in memory constrained environments. The bottom-up state spill strategy is an operator-level strategy that treats all data in one...

• Type: Conference
• Year: 2006
• Where: SIGMOD
• Authors: Bin Liu, Yali Zhu, Elke A. Rundensteiner
• Keywords: Data Spill Strategies, Database, SIGMOD 2006, State Spill Strategies, State Spill Strategy