Performance thresholding in practical text classification

In practical classification, there is often a mix of learnable and unlearnable classes and only a classifier above a minimum performance threshold can be deployed. This problem is exacerbated if the training set is created by active learning. The bias of actively learned training sets makes it hard to determine whether a class has been learned. We give evidence that there is no general and efficient method for reducing the bias and correctly identifying classes that have been learned. However, we characterize a number of scenarios where active learning can succeed despite these difficulties. Categories and Subject Descriptors I.2.6 [Artificial Intelligence]: Learning; H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing General Terms Algorithms, Measurement, Performance, Experimentation Keywords practical text classification, active learning, accuracy estimation, learnability

• Type: Conference
• Year: 2006
• Where: CIKM
• Authors: Hinrich Schütze, Emre Velipasaoglu, Jan O. Pedersen
• Keywords: Active Learning, CIKM 2006, Information Management, Minimum Performance Threshold, Training Sets