Hybrid AI approaches for context recognition: application to activity recognition and anticipation and context abnormalities handling in Ambient Intelligence environments
Approches d'IA hybrides pour la reconnaissance du contexte : application à la reconnaissance et à l'anticipation d'activités, et à la gestion des anomalies de contexte dans les environnements d'Intelligence Ambiante
Résumé
Ambient Intelligence (AmI) systems aim to provide users with assistance services intended
to improve the quality of their lives in terms of autonomy, safety, and well-being. The
design of AmI systems capable of accurate, fine-grained and consistent recognition of
the spatial and/or temporal user’s context, taking into account the uncertainty and partial
observability of AmI environments, poses several challenges to enable a better adaptation
of the assistance services to the user’s context. The purpose of this thesis is to propose
a set of contributions that address these challenges. Firstly, a descriptive and narrative
context ontology is proposed to model contextual knowledge in AmI environments. The
purpose of this ontology is the modeling of the user’s context taking into account different
context attributes and defining axioms of the commonsense reasoning necessary to infer and
update the context of the user. In contrast to state-of-the-art ontologies, the proposed context
ontology includes (i) a TBox representing the core domain ontology defined by concepts
and relations, (ii) an ABox of propositional formulas corresponding to context attribute
instantiations, and (iii) an RBox, represented by an ASP logical program, consisting of rule
templates, such as specification of the effects of events, specification of triggered events,
aggregation of context components, and planning of sensing and assistance actions. The
TBox, ABox, and RBox represent a basis of the frameworks developed in this thesis and
play a crucial role in improving the recognition of the user’s context. The second contribution is an ontology-based hybrid framework that combine probabilistic commonsense
reasoning and probabilistic planning to recognize the user’s context, in particular, context
abnormalities, and provide context-aware assistance services, in presence of uncertainty and partial observability of the environments. This framework exploits context attribute
predictions, namely user’s activity and user’s location, provided by deep learning models. In
this framework, the probabilistic commonsense reasoning is based on the proposed context
ontology to define the axiomatization of the context inference and planning under uncertainty. Probabilistic planning is used to characterize abnormal context by coping with the
incompleteness of contextual knowledge due to the partial observability of AmI environments. In addition, probabilistic planning allows to adapt assistance services provided to the
user according to its context. The proposed framework was evaluated using transformers
and CNN-LSTM models considering Orange4Home and SIMADL datasets. The results
show the effectiveness of the framework to recognize user’s contexts, in terms of user’s
activity and location, along with context abnormalities in uncertain and partially observable
environments. Thirdly, a hybrid framework combining deep learning and probabilistic
commonsense reasoning for anticipating human activities based on egocentric videos is
proposed. The probabilistic commonsense reasoning exploited in this framework is based on
abductive reasoning to anticipate both human atomic and composite activities, and temporal
reasoning to capture context attribute changes. Deep learning models, namely YOLOv5 and
ResNet, were exploited to recognize context attributes, such as objects, human hands, and
human locations. The context ontology is used to model the relationships between atomic
activities and composite activities. The evaluation of the framework shows its ability to
anticipate composite activities over a time horizon of minutes, in contrast to state-of-the-art
approaches that can only anticipate atomic activities over a time horizon of seconds. It also
showed good performance in terms of accuracy of classification of anticipated activities and
computation time. Lastly, a stream reasoning-based framework is proposed to anticipate
atomic and composite human activities from data streams of context attributes collected
on-the-fly. YOLOv7 and ResNet deep learning models were used to recognize context
attributes, such as objects used in activities, hands and user locations. The stream reasoning
system performs causal, abductive and temporal reasoning with contextual knowledge obtained at run-time. Dynamic effect axioms were introduced to anticipate composite activities
that can be subject to unforeseen events, such as skipping an atomic activity and delay an
atomic activity. The proposed framework was validated through experiments conducted
in a kitchen environment. The remarkably high performance in terms of the number of
activity anticipations shows the ability of the framework to take into account the contextual
knowledge of past episodes needed to anticipate composite activities. The performance in
terms of contextual knowledge inference time indicates that the framework is suitable for
real-world applications.
Ambient Intelligence (AmI) systems aim to provide users with assistance services intended
to improve the quality of their lives in terms of autonomy, safety, and well-being. The
design of AmI systems capable of accurate, fine-grained and consistent recognition of
the spatial and/or temporal user’s context, taking into account the uncertainty and partial
observability of AmI environments, poses several challenges to enable a better adaptation
of the assistance services to the user’s context. The purpose of this thesis is to propose
a set of contributions that address these challenges. Firstly, a descriptive and narrative
context ontology is proposed to model contextual knowledge in AmI environments. The
purpose of this ontology is the modeling of the user’s context taking into account different
context attributes and defining axioms of the commonsense reasoning necessary to infer and
update the context of the user. In contrast to state-of-the-art ontologies, the proposed context
ontology includes (i) a TBox representing the core domain ontology defined by concepts
and relations, (ii) an ABox of propositional formulas corresponding to context attribute
instantiations, and (iii) an RBox, represented by an ASP logical program, consisting of rule
templates, such as specification of the effects of events, specification of triggered events,
aggregation of context components, and planning of sensing and assistance actions. The
TBox, ABox, and RBox represent a basis of the frameworks developed in this thesis and
play a crucial role in improving the recognition of the user’s context. The second contribution is an ontology-based hybrid framework that combine probabilistic commonsense
reasoning and probabilistic planning to recognize the user’s context, in particular, context
abnormalities, and provide context-aware assistance services, in presence of uncertainty and partial observability of the environments. This framework exploits context attribute
predictions, namely user’s activity and user’s location, provided by deep learning models. In
this framework, the probabilistic commonsense reasoning is based on the proposed context
ontology to define the axiomatization of the context inference and planning under uncertainty. Probabilistic planning is used to characterize abnormal context by coping with the
incompleteness of contextual knowledge due to the partial observability of AmI environments. In addition, probabilistic planning allows to adapt assistance services provided to the
user according to its context. The proposed framework was evaluated using transformers
and CNN-LSTM models considering Orange4Home and SIMADL datasets. The results
show the effectiveness of the framework to recognize user’s contexts, in terms of user’s
activity and location, along with context abnormalities in uncertain and partially observable
environments. Thirdly, a hybrid framework combining deep learning and probabilistic
commonsense reasoning for anticipating human activities based on egocentric videos is
proposed. The probabilistic commonsense reasoning exploited in this framework is based on
abductive reasoning to anticipate both human atomic and composite activities, and temporal
reasoning to capture context attribute changes. Deep learning models, namely YOLOv5 and
ResNet, were exploited to recognize context attributes, such as objects, human hands, and
human locations. The context ontology is used to model the relationships between atomic
activities and composite activities. The evaluation of the framework shows its ability to
anticipate composite activities over a time horizon of minutes, in contrast to state-of-the-art
approaches that can only anticipate atomic activities over a time horizon of seconds. It also
showed good performance in terms of accuracy of classification of anticipated activities and
computation time. Lastly, a stream reasoning-based framework is proposed to anticipate
atomic and composite human activities from data streams of context attributes collected
on-the-fly. YOLOv7 and ResNet deep learning models were used to recognize context
attributes, such as objects used in activities, hands and user locations. The stream reasoning
system performs causal, abductive and temporal reasoning with contextual knowledge obtained at run-time. Dynamic effect axioms were introduced to anticipate composite activities
that can be subject to unforeseen events, such as skipping an atomic activity and delay an
atomic activity. The proposed framework was validated through experiments conducted
in a kitchen environment. The remarkably high performance in terms of the number of
activity anticipations shows the ability of the framework to take into account the contextual
knowledge of past episodes needed to anticipate composite activities. The performance in
terms of contextual knowledge inference time indicates that the framework is suitable for
real-world applications.