A novel system for intelligent visual tracking focus on the unknown object tracking in short or mid-term video stream of the object in unconstrained environment. Also it commonly focuses only few kind of objects for doing definite programmed mission like moving the object or opening the door. It means that many vision tracking did not consider the general category of the object. For more intelligent visual tracking system, long-term robust unknown object tracking and recognizer is important. We propose the real-time intelligent tracking system equipped with recognizer operating on long-term situation by using novel tracking framework TrackingLearning-Detection (TLD) and deep convolutional neural network. The system is validated on long-term surveillance to tracking and recognizing object. Categories and Subject Descriptors