Efficient video content management and exploitation requires extraction of the underlying semantics, a non-trivial task associating low-level features of the image domain and high-level semantic descriptions. In this paper, a knowledge-assisted approach for extracting semantics of domain-specific video content is presented. Domain knowledge considers both low-level features (color, motion, shape) and spatial behavior (topological and directional information). First a segmentation algorithm generates a set of oversegmented, homogenous atom-regions. A genetic algorithm is applyed, after the preprocessing step, in order to optimize the scene interpretation according to the knowledge of the specific domain. The proposed approach was tested on the Tennis and Formula One domains and shows promising results.