Most recent research work on sports video analysis focuses on broadcast video. The broadcast video is a post-produced video and has much additional editing information inserted. In this paper, we propose a novel event detection framework in sports game only based on the video taken by single (main) camera. Compared with event detection from broadcast video, the proposed framework is more challenging as interesting events needs to be automatically detected. The results of the research could be used for automatic replay generation in sports video. In this paper, a mid-level representation is first created based on low-level audio/visual feature and events related to the replay are then detected from this representation. We experiment our framework and some promising results are obtained.