Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization

Abstract. The optimization of hyperparameters is often done manually or exhaustively but recent work has shown that automatic methods can optimize hyperparameters faster and even achieve better nal performance. Sequential model-based optimization (SMBO) is the current state of the art framework for automatic hyperparameter optimization. Currently, it consists of three components: a surrogate model, an acquisition function and an initialization technique. We propose to add a fourth component, a way of pruning the hyperparameter search space which is a common way of accelerating the search in many domains but yet has not been applied to hyperparameter optimization. We propose to discard regions of the search space that are unlikely to contain better hyperparameter congurations by transferring knowledge from past experiments on other data sets as well as taking into account the evaluations already done on the current data set. Pruning as a new component for SMBO is an orthogonal contrib...

• Type: Journal
• Year: 2015
• Where: PKDD
• Authors: Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme
• Keywords: Data Mining, PKDD 2015