In this paper, we introduce a novel face detection and recognition system based on deep convolutional networks, designed to improve the visually impaired users' interaction and communication in social encounters. A first feature of the proposed architecture concerns a face detection system able to identify various persons existent in the scene regardless of the subject location or pose. Then, the faces are tracked between successive frames using a CNN (Convolutional Neural Networks)-based tracker trained offline with generic motion patterns. The system can handle face occlusion, rotation or position pose variation, as well as and important illumination changes. Finally, the faces are recognized, in real-time, directly from the video stream. The major contribution of the paper consists in a novel weight adaptation scheme able to determine the relevance of face instances and to create a global, fixed-size representation from all face instances tracked during the video stream, while remaining independent of the track length. The experimental evaluation is performed on a set of 30 video elements acquired with the help of visually impaired users. The tests experimental results obtained validate the approach with average detection and recognition scores superior to 85%