Hybrid AI approaches for context recognition: application to activity recognition and anticipation and context abnormalities handling in Ambient Intelligence environments - Archive ouverte HAL Accéder directement au contenu
Thèse Année : 2023

Hybrid AI approaches for context recognition: application to activity recognition and anticipation and context abnormalities handling in Ambient Intelligence environments

Approches d'IA hybrides pour la reconnaissance du contexte : application à la reconnaissance et à l'anticipation d'activités, et à la gestion des anomalies de contexte dans les environnements d'Intelligence Ambiante

Résumé

Ambient Intelligence (AmI) systems aim to provide users with assistance services intended to improve the quality of their lives in terms of autonomy, safety, and well-being. The design of AmI systems capable of accurate, fine-grained and consistent recognition of the spatial and/or temporal user’s context, taking into account the uncertainty and partial observability of AmI environments, poses several challenges to enable a better adaptation of the assistance services to the user’s context. The purpose of this thesis is to propose a set of contributions that address these challenges. Firstly, a descriptive and narrative context ontology is proposed to model contextual knowledge in AmI environments. The purpose of this ontology is the modeling of the user’s context taking into account different context attributes and defining axioms of the commonsense reasoning necessary to infer and update the context of the user. In contrast to state-of-the-art ontologies, the proposed context ontology includes (i) a TBox representing the core domain ontology defined by concepts and relations, (ii) an ABox of propositional formulas corresponding to context attribute instantiations, and (iii) an RBox, represented by an ASP logical program, consisting of rule templates, such as specification of the effects of events, specification of triggered events, aggregation of context components, and planning of sensing and assistance actions. The TBox, ABox, and RBox represent a basis of the frameworks developed in this thesis and play a crucial role in improving the recognition of the user’s context. The second contribution is an ontology-based hybrid framework that combine probabilistic commonsense reasoning and probabilistic planning to recognize the user’s context, in particular, context abnormalities, and provide context-aware assistance services, in presence of uncertainty and partial observability of the environments. This framework exploits context attribute predictions, namely user’s activity and user’s location, provided by deep learning models. In this framework, the probabilistic commonsense reasoning is based on the proposed context ontology to define the axiomatization of the context inference and planning under uncertainty. Probabilistic planning is used to characterize abnormal context by coping with the incompleteness of contextual knowledge due to the partial observability of AmI environments. In addition, probabilistic planning allows to adapt assistance services provided to the user according to its context. The proposed framework was evaluated using transformers and CNN-LSTM models considering Orange4Home and SIMADL datasets. The results show the effectiveness of the framework to recognize user’s contexts, in terms of user’s activity and location, along with context abnormalities in uncertain and partially observable environments. Thirdly, a hybrid framework combining deep learning and probabilistic commonsense reasoning for anticipating human activities based on egocentric videos is proposed. The probabilistic commonsense reasoning exploited in this framework is based on abductive reasoning to anticipate both human atomic and composite activities, and temporal reasoning to capture context attribute changes. Deep learning models, namely YOLOv5 and ResNet, were exploited to recognize context attributes, such as objects, human hands, and human locations. The context ontology is used to model the relationships between atomic activities and composite activities. The evaluation of the framework shows its ability to anticipate composite activities over a time horizon of minutes, in contrast to state-of-the-art approaches that can only anticipate atomic activities over a time horizon of seconds. It also showed good performance in terms of accuracy of classification of anticipated activities and computation time. Lastly, a stream reasoning-based framework is proposed to anticipate atomic and composite human activities from data streams of context attributes collected on-the-fly. YOLOv7 and ResNet deep learning models were used to recognize context attributes, such as objects used in activities, hands and user locations. The stream reasoning system performs causal, abductive and temporal reasoning with contextual knowledge obtained at run-time. Dynamic effect axioms were introduced to anticipate composite activities that can be subject to unforeseen events, such as skipping an atomic activity and delay an atomic activity. The proposed framework was validated through experiments conducted in a kitchen environment. The remarkably high performance in terms of the number of activity anticipations shows the ability of the framework to take into account the contextual knowledge of past episodes needed to anticipate composite activities. The performance in terms of contextual knowledge inference time indicates that the framework is suitable for real-world applications.
Ambient Intelligence (AmI) systems aim to provide users with assistance services intended to improve the quality of their lives in terms of autonomy, safety, and well-being. The design of AmI systems capable of accurate, fine-grained and consistent recognition of the spatial and/or temporal user’s context, taking into account the uncertainty and partial observability of AmI environments, poses several challenges to enable a better adaptation of the assistance services to the user’s context. The purpose of this thesis is to propose a set of contributions that address these challenges. Firstly, a descriptive and narrative context ontology is proposed to model contextual knowledge in AmI environments. The purpose of this ontology is the modeling of the user’s context taking into account different context attributes and defining axioms of the commonsense reasoning necessary to infer and update the context of the user. In contrast to state-of-the-art ontologies, the proposed context ontology includes (i) a TBox representing the core domain ontology defined by concepts and relations, (ii) an ABox of propositional formulas corresponding to context attribute instantiations, and (iii) an RBox, represented by an ASP logical program, consisting of rule templates, such as specification of the effects of events, specification of triggered events, aggregation of context components, and planning of sensing and assistance actions. The TBox, ABox, and RBox represent a basis of the frameworks developed in this thesis and play a crucial role in improving the recognition of the user’s context. The second contribution is an ontology-based hybrid framework that combine probabilistic commonsense reasoning and probabilistic planning to recognize the user’s context, in particular, context abnormalities, and provide context-aware assistance services, in presence of uncertainty and partial observability of the environments. This framework exploits context attribute predictions, namely user’s activity and user’s location, provided by deep learning models. In this framework, the probabilistic commonsense reasoning is based on the proposed context ontology to define the axiomatization of the context inference and planning under uncertainty. Probabilistic planning is used to characterize abnormal context by coping with the incompleteness of contextual knowledge due to the partial observability of AmI environments. In addition, probabilistic planning allows to adapt assistance services provided to the user according to its context. The proposed framework was evaluated using transformers and CNN-LSTM models considering Orange4Home and SIMADL datasets. The results show the effectiveness of the framework to recognize user’s contexts, in terms of user’s activity and location, along with context abnormalities in uncertain and partially observable environments. Thirdly, a hybrid framework combining deep learning and probabilistic commonsense reasoning for anticipating human activities based on egocentric videos is proposed. The probabilistic commonsense reasoning exploited in this framework is based on abductive reasoning to anticipate both human atomic and composite activities, and temporal reasoning to capture context attribute changes. Deep learning models, namely YOLOv5 and ResNet, were exploited to recognize context attributes, such as objects, human hands, and human locations. The context ontology is used to model the relationships between atomic activities and composite activities. The evaluation of the framework shows its ability to anticipate composite activities over a time horizon of minutes, in contrast to state-of-the-art approaches that can only anticipate atomic activities over a time horizon of seconds. It also showed good performance in terms of accuracy of classification of anticipated activities and computation time. Lastly, a stream reasoning-based framework is proposed to anticipate atomic and composite human activities from data streams of context attributes collected on-the-fly. YOLOv7 and ResNet deep learning models were used to recognize context attributes, such as objects used in activities, hands and user locations. The stream reasoning system performs causal, abductive and temporal reasoning with contextual knowledge obtained at run-time. Dynamic effect axioms were introduced to anticipate composite activities that can be subject to unforeseen events, such as skipping an atomic activity and delay an atomic activity. The proposed framework was validated through experiments conducted in a kitchen environment. The remarkably high performance in terms of the number of activity anticipations shows the ability of the framework to take into account the contextual knowledge of past episodes needed to anticipate composite activities. The performance in terms of contextual knowledge inference time indicates that the framework is suitable for real-world applications.
Fichier non déposé

Dates et versions

tel-04328398 , version 1 (07-12-2023)

Identifiants

  • HAL Id : tel-04328398 , version 1

Citer

Koussaila Moulouel. Hybrid AI approaches for context recognition: application to activity recognition and anticipation and context abnormalities handling in Ambient Intelligence environments. Artificial Intelligence [cs.AI]. Université Paris Est Créteil, 2023. English. ⟨NNT : ⟩. ⟨tel-04328398⟩

Collections

LISSI UPEC
30 Consultations
0 Téléchargements

Partager

Gmail Facebook X LinkedIn More