Distracted driving is one of the main causes of deaths and injuries in the world. Monitoring driver behaviors through Driver Action Recognition (DAR) contributes significantly to building safer transportation systems. However, in naturalistic driving settings, this task is complex and challenging because of numerous difficulties, such as high illumination variation and cluttered and dynamic background. In this paper, we introduce a novel hard attention network that highlights the most pertinent driving-scene information while filtering out irrelevant data. Specifically, only local discriminative salient regions are exploited through a hard attention mechanism. The experimental results indicate that our approach significantly enhances DAR performance. We evaluated our network on three diverse state-of-the-art datasets recorded in real-world conditions: it achieves up to 95.83% in terms of safe driving recognition and up to 99.07% in terms of distraction detection. The proposed approach outperforms the soft attention-based DAR not only in detection and recognition performance but also in computation complexity by 38.71% less runtime. For reproducible research, the code is available at https://github.com/JEGHAMI/HSA