In this paper we introduce a novel single object tracker based on two convolutional neural networks (CNNs) trained offline using data from large videos repositories. The key principle consists of alternating between tracking using motion information and adjusting the predicted location based on visual similarity. First, we construct a deep regression network architecture able to learn generic relations between the object appearance models and its associated motion patterns. Then, based on visual similarity constraints, the objects bounding box position, size and shape are continuously updated in order to maximize a patch similarity function designed using CNN. Finally, a multi-resolution fusion between the outputs of the two CNNs is performed for accurate object localization. The experimental evaluation performed on challenging datasets, proposed in the visual object tracking (VOT) international contest, validates the proposed method when compared with state-of-theart systems. In terms of computational speed our tracker runs at 20fps