Online Incremental Feature Learning with Denoising Autoencoders

While determining model complexity is an important problem in machine learning, many feature learning algorithms rely on cross-validation to choose an optimal number of features, which is usually challenging for online learning from a massive stream of data. In this paper, we propose an incremental feature learning algorithm to determine the optimal model complexity for large-scale, online datasets based on the denoising autoencoder. This algorithm is composed of two processes: adding features and merging features. Specifically, it adds new features to minimize the objective function’s residual and merges similar features to obtain a compact feature representation and prevent over-fitting. Our experiments show that the proposed model quickly converges to the optimal number of features in a large-scale online setting. In classification tasks, our model outperforms the (non-incremental) denoising autoencoder, and deep networks constructed from our algorithm perform favorably compar...

• Type: Journal
• Year: 2012
• Where: JMLR
• Authors: Guanyu Zhou, Kihyuk Sohn, Honglak Lee
• Keywords: Belief Networks, Cross Validation, JMLR 2012, Model Complexity, Programming Languages