The fast growth of urban populations across evolving cities is resulting in new types of technical, physical, material, and safety-related challenges and constraints (Chen et al. 2019). Humans perceive, move, and perform actions (called behaviors) in different urban areas, and make interactions with the objects, the other humans, and the environment for achieving their goals for attaining the desired level of satisfaction (Wagner et al., 2018; Chen et al., 2015). More formally, human behaviors are observable actions or reactions of humans in response to external or internal stimuli (Wagner et al., 2018; Hong et al., 2015; Yan et al., 2015). Over the last decade, human behavior understanding has attracted the strong interest of the researchers (Wagner et al., 2018) for better monitoring and controlling the facilities in urban areas. This massive interest in behavior understanding is because of its numerous potential applications which can be divided into three major domains as shown in Fig. 1.