Recognition using appearance features is confounded by
phenomena that cause images of the same object to look different,
or images of different objects to look the same. This
may occur because the same object looks different from different
viewing directions, or because two generally different
objects have views from which they look similar. In this
paper, we introduce the idea of discriminative aspect, a set
of latent variables that encode these phenomena. Changes
in view direction are one cause of changes in discriminative
aspect, but others include changes in texture or lighting.
However, images are not labelled with relevant discriminative
aspect parameters. We describe a method to
improve discrimination by inferring and then using latent
discriminative aspect parameters. We apply our method to
two parallel problems: object category recognition and human
activity recognition. In each case, appearance features
are powerful given appropriate training data, but traditi...