In recent years, real-time human activity recognition (HAR) has reached importance due to its applications in various domains such as assistive services for the elderly in smart buildings, monitoring, well-being, comfort and security. Various techniques, researched within the image processing and computer vision communities, have been established to recognize human activities in real-time, but all of them are based on wearable sensors and there is no much attention for ambient sensor based approaches. In the literature, deep learning (DL) is one of effective and cost-efficient supervised learning model and different architectures haves been investigated for real-time HAR. However, it still struggles with the quality of data as well as hardware implementation issues. This paper presents two contributions. Firstly, an intensive analysis of DL architectures and its characteristics along with their limitations in the framework of real time HAR are investigated. Secondly, existing hardware architectures and related challenges in this field are highlighted (adaptation of DL architectures towards microcontrollers, difficulty to provide a smart home with numerous sensors and trends regarding cloud-bases approaches). Then, new research directions and solutions around the real-time data quality assessment, the study of main performance factors for DL on microcontrollers, the concept of minimal sensors set up for the employment of IoT devices and the distributed intelligence are suggested to solve them respectively and to improve this field.