Feature Subset Selection Using the Wrapper Method: Overfitting and Dynamic Search Space Topology

In the wrapperapproachto feature subset selection, a searchfor an optimalset of features is madeusingthe induction algorithm as a black box. Theestimated future performanceof the algorithm is the heuristic guiding the search. Statistical methodsfor feature subset selection includingforwardselection, backward elimination, and their stepwisevariants can be viewed as simplehill-climbing techniquesin the spaceof feature subsets. Weutilize best-first searchto find a good feature subset and discuss overfitting problemsthat maybe associated with searching too manyfeature subsets. Weintroduce compoundoperators that dynamically changethe topologyof the search space to better utilize the informationavailable fromthe evaluation of feature subsets. Weshow that compound operators unify previous approaches that deal with relevant and irrelevant features. Theimprovedfeature subset selection yields significant improvements for real-world datasets whenusing the ID3and the Naive-Bayesinduction algorith...

• Type: Conference
• Year: 1995
• Where: KDD
• Authors: Ron Kohavi, Dan Sommerfield
• Keywords: Data Mining, Feature Subset Selection, Feature Subsets, KDD 1995, Spaceof Feature Subsets