Driver behaviors and decisions are crucial factors for on-road driving safety. With a precise driver behavior monitoring system, traffic accidents and injuries can be significantly reduced. However, understanding human behaviors in real-world driving settings is a challenging task because of the uncontrolled conditions including illumination variation, occlusion, and dynamic and cluttered background. In this paper, a Kinect sensor, which provides multimodal signals, is adopted as a driver monitoring sensor to recognize safe driving and common secondary most distracting in-vehicle actions. We propose a novel soft spatial attention-based network named the Depth-based Spatial Attention network (DSA), which adds a cognitive process to deep network by selectively focusing on the driver's silhouette and motion in the cluttered driving scene. In fact, at each time t, we introduce a new weighted RGB frame based on an attention model designed using a depth frame. The final classification accuracy is substantially enhanced compared to the state-of-the-art results with an achieved improvement of up to 27%. © 2001-2012 IEEE.