Abstract. This paper addresses the issue in scenario-based understanding of human behavior from videos in a nursing care monitoring system. The analysis is carried out based on experiments consisting of single-state scenarios and multi-state scenarios where the former monitors activities under contextual contexts for elementary behavior reasoning, while the latter dictating the elementary behavior order for behavior reasoning, with a priori knowledge in system profile for normality detection. By integrating the activities, situation context, and profile knowledge we can have a better understanding of patients in a monitoring system. In activity recognition, a Negation-Selection mechanism is developed. which employs a divide-and-conquer concept with the Negation using posture transition to preclude the negative set from the activities. The Selection that follows the Negation uses a moving history trace for activity recognition. Such a history trace composes not only the pose from sing...