Multimodal Vision Transformers with Forced Attention for Behavior Analysis - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Multimodal Vision Transformers with Forced Attention for Behavior Analysis

Résumé

Human behavior understanding requires looking at minute details in the large context of a scene containing multiple input modalities. It is necessary as it allows the design of more human-like machines. While transformer approaches have shown great improvements, they face multiple challenges such as lack of data or background noise. To tackle these, we introduce the Forced Attention (FAt) Transformer which utilize forced attention with a modified backbone for input encoding and a use of additional inputs. In addition to improving the performance on different tasks and inputs, the modification requires less time and memory resources. We provide a model for a generalised feature extraction for tasks concerning social signals and behavior analysis. Our focus is on understanding behavior in videos where people are interacting with each other or talking into the camera which simulates the first person point of view in social interaction. FAt Transformers are applied to two downstream tasks: personality recognition and body language recognition. We achieve state-of-the-art results for Udiva v0.5, First Impressions v2 and MPII Group Interaction datasets. We further provide an extensive ablation study of the proposed architecture.
Fichier principal
Vignette du fichier
WACV_2023__Multimodal_Vision_Transformers_with_Forced_Attention_for_Behavior_Analysis.pdf (409.67 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : Domaine public

Dates et versions

hal-03936484 , version 1 (12-01-2023)

Licence

Domaine public

Identifiants

Citer

Tanay Agrawal, Michal Balazia, Philipp Müller, Francois F Bremond. Multimodal Vision Transformers with Forced Attention for Behavior Analysis. WACV '23: IEEE International Winter Conference on Applications in Computer Vision, IEEE, Jan 2023, Waikoloa, United States. ⟨hal-03936484⟩
39 Consultations
60 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More