We propose using simple mixture models to define a set of mid-level binary local features based on binary oriented edge input. The features capture natural local structures in the data and yield very high classification rates when used with a variety of classifiers trained on small training sets, exhibiting robustness to degradation with clutter. Of particular interest are the use of the features as variables in simple statistical models for the objects thus enabling likelihood based classification. Pre-training decision boundaries between classes, a necessary component of non-parametric techniques, is thus avoided. Class models are trained separately with no need to access data of other classes. Experimental results are presented for handwritten character recognition, classification of deformed LATEX symbols involving hundreds of classes, and side view car detection.