Social network contents are not limited to text but also multimedia. Dailymotion, YouTube, and MySpace are examples of successful sites which allow users to share videos among themselves. Due to the huge amount of videos, grouping videos with similar contents together can help users to search videos more efficiently. Unlike the traditional approach to group videos into some predefined categories, we propose a novel comment-based matrix factorization technique to categorize videos and generate concept words to facilitate searching and indexing. Since the categorization is learnt from users feedback, it can accurately represent the user sentiment on the videos. Experiments conducted by using empirical data collected from YouTube shows the effectiveness of our proposed methodologies.