Dot-Product Join: An Array-Relation Join Operator for Big Model Analytics

Big Data analytics has been approached exclusively from a data-parallel perspective, where data are partitioned to multiple workers – threads or separate servers – and model training is executed concurrently over different partitions, under various synchronization schemes that guarantee speedup and/or convergence. The dual – Big Model – problem that, surprisingly, has received no attention in database analytics, is how to manage models with millions if not billions of parameters that do not fit in memory. This distinction in model representation changes fundamentally how in-database analytics tasks are carried out. In this paper, we introduce the first secondary storage array-relation dot-product join operator between a set of sparse arrays and a dense relation targeted. The paramount challenge in designing such an operator is how to optimally schedule access to the dense relation based on the sparse non-contiguous entries in the sparse arrays. We prove that this problem is ...

• Type: Journal
• Year: 2016
• Where: CORR
• Authors: Chengjie Qin, Florin Rusu
• Keywords: CORR 2016, Education