Facial feature tracking is a crucial and challenging task in computer vision. Recently online-learning methods have become increasingly popular on account of their strong ability to adapt to variations and have achieved good results in tracking. However, all previous work used only raw intensity to build the model, which is very sensitive to condition changes. In this work, we present a real time, fully automatic facial feature detection and tracking approach using adaptive observation models based on edge structure, which is more reliable especially when the lighting state alters during tracking. Experimental results demonstrate that using edge map measures in observation modeling can improve the accuracy and robustness of tracking.