— This paper describes a panoramic view-based navigation in outdoor environments. We have been developing a two-phase navigation method. In the training phase, the robot acquires image sequences along the desired route and automatically learns the route visually. In the subsequent autonomous navigation phase, the robot moves by localizing itself by comparing input images with the learned route representation. To be robust to changes of weather and seasons, an object-based comparison is adopted. Our previous method applied a support vector machine (SVM) algorithm to object recognition and localization and exhibited a satisfactory performance but was sometimes sensitive to the variation of the robot’s heading. This paper thus extends the method to use panoramic images. By searching the image for the region which matches the model image the most, a new method can considerably improve the localization performance and provide the robot with globally correct directions to move.