DEDUCE: at the intersection of MapReduce and stream processing

MapReduce and stream processing are two emerging, but different, paradigms for analyzing, processing and making sense of large volumes of modern day data. While MapReduce offers the capability to analyze several terabytes of stored data, stream processing solutions offer the ability to process, possibly, a few million updates every second. However, there is an increasing number of data processing applications which need a solution that effectively and efficiently combines the benefits of MapReduce and stream processing to address their data processing needs. For example, in the automated stock trading domain, applications usually require periodic analysis of large amounts of stored data to generate a model using MapReduce, which is then used to process a stream of incident updates using a stream processing system. This paper presents DEDUCE, which extends IBM's System S stream processing middleware with support for MapReduce by providing (1) language and runtime support for easil...

• Type: Conference
• Year: 2010
• Where: EDBT
• Authors: Vibhore Kumar, Henrique Andrade, Bugra Gedik, Kun-Lung Wu
• Keywords: Data Processing, Database, EDBT 2010, Stream Processing, Stream Processing Solutions