Digital video databases have become more pervasive and finding video clips quickly in large databases becomes a major challenge. Due to the nature of video, accessing contents of video is difficult and time-consuming. With content-based video systems today, there exists a significant gap between the user's information and what the system can deliver. Therefore, enabling intelligent means of interpretation on visual content, semantics annotation and retrieval are important topics of research. In this paper, we consider semantic interpretation of the contents as annotation tags for video clips, giving a retrieval-driven and applicationoriented semantics extraction, annotation and retrieval model for video database management system. This system design employs an algorithm on objects' relation and it can reveal the semantics defined with fast real-time computation.