In this paper we introduce a real-time obstacle recognition framework designed to alert the visually impaired people/blind of their presence and to assist humans to navigate safely, in indoor and outdoor environments, by handling a Smartphone device. Static and dynamic objects are detected using interest points selected based on an image grid and tracked using the multiscale Lucas-Kanade algorithm. Next, we activated an object classification methodology. We incorporate HOG (Histogram of Oriented Gradients) descriptor into the BoVW (Bag of Visual Words) retrieval framework and demonstrate how this combination may be used for obstacle classification in video streams. The experimental results performed on various challenging scenes demonstrate that our approach is effective in image sequence with important camera movement, including noise and low resolution data and achieves high accuracy, while being computational efficient.