We propose use of an appearance manifold with embedded covariance matrix as a technique for recognizing 3D objects from images that are influenced by geometric and quality-degraded effects. Our strategy covers the construction of this appearance manifold by giving consideration to pose changes. In the proposed method, the correspondence of each learning pose is not based on the eigenpoint but directly from the covariance matrix. Thus, we eliminate the dependency on eigenpoint-to-eigenpoint correspondence, which is the main cause of misclassification due to the phenomenon of the eigenpoint’s shifting position. Experimental results show that our approach achieves higher recognition accuracies than using a simple appearance manifold. Consequently, it can provide a more efficient way of developing a robust 3D object recognition system.