Identifying other persons is a major challenge for blinds and visually impaired people that can hinder interaction in social activities. This paper introduces a novel CNN based face detection, tacking and recognition system designed to improve users' interaction and communication in social encounters. The major contribution consists in a novel weight adaptation scheme able to determine the relevance of face instances and to create a global, fixed-size representation from all face instances tracked during the video stream, while remaining independent of the track length. The experimental evaluation performed on a large set of video streams validates the approach that returns accuracy and recognition rates superior to 90%.