This paper presents a novel background modeling approach for day and night video surveillance. A great number of background models have been proposed to represent the background scene for video surveillance. In this paper, we propose a novel background modeling approach by using the phase space trajectory to represent the change of intensity over time for each pixel. If the intensity of a pixel which originally belongs to the background deviates from the original trajectory in phase space, then it is considered a foreground object pixel. In this manner, we are able to separate the foreground object from the background scene easily. The experimental results show the feasibility of the proposed background model.