In this paper we present a scene-based video watermarking scheme using support vector machines (SVMs). In a given scene, the algorithm uses the first h′ frames to train an embedding SVM, and uses this SVM to watermark the rest of frames. In the extracting phrase, the detector uses center h frames of the first h′ frames to train an extracting SVM. The final extracted watermark in a given scene is the average of watermarks extracted from the remaining frames. Watermarks are embedded in l longest scenes of a video such that it is time efficient and capable to resist possible frames swapping/deleting /duplicating attacks. Collusion attacks on a watermarked video are examined. The proposed algorithm is shown to be robust to compression and collusion attacks, and it has novelty and practicability on SVM-applications.