The application of semi-supervised learning algorithms to large scale vision problems suffers from the bad scaling behavior of most methods. Based on the Expectation Regularization principle, in this paper we propose a novel semi-supervised boosting method, called SERBoost that can be applied to large scale vision problems and its complexity is dominated by the base learners. The algorithm provides a margin regularizer for the boosting cost function and shows a principled way of utilizing prior knowledge. We demonstrate the performance of SERBoost on the Pascal VOC2006 set and compare it to other supervised and semisupervised methods, where SERBoost shows improvements both in terms of classification accuracy and computational speed.