Driver Distraction is, increasingly, one of the major causes of road accidents. Distractions can be caused by activities that may shift the driver's attention and potentially evoke negative emotional states. Recently, there has been notable interest in Driver Assistance Systems (DAS) designed for Driver Distraction Detection (DDD). These systems focus on improving both safety and driver comfort by issuing alerts for potential hazards. Recent advancements in DAS have prominently incorporated deep learning techniques, showcasing a shift towards sophisticated and intelligent approaches for enhanced performance and functionality. However, model architecture design is mainly based on expert knowledge and empirical evaluations, which are time-consuming and resource-intensive. Hence, it is hard to design a model that is both efficient and accurate at the same time. This paper presents a Neural Architecture Search (NAS)-based approach for efficient deep CNN design for DDD. The proposed approach leverages RGB images to train a lightweight model with few parameters and high recognition accuracy. Experimental validation is performed on two driver distraction benchmark datasets, demonstrating that the proposed model outperforms state-of-the-art models in terms of efficiency while maintaining competitive accuracy. We report 99.08% and 93.23% with model parameter numbers equal to 0.10 and 0.14 Million parameters for respectively SFD and AUC datasets. The obtained architectures are both accurate and lightweight for DDD.