Article Dans Une Revue Cognitive Computation Année : 2025

Multimodal Framework for Automatic Behavior Analysis of Children with Autism During ADOS-2

Kaan Karaköse
  • Fonction : Auteur
Ádám Fodor
  • Fonction : Auteur
Linyun Xiang
  • Fonction : Auteur
Viktor Varga
  • Fonction : Auteur
Latha Soorya
  • Fonction : Auteur
Emily Dillon
  • Fonction : Auteur
Péter Kun
  • Fonction : Auteur
András Sárkány
  • Fonction : Auteur
Kristian Fenech
  • Fonction : Auteur
András Lőrincz
  • Fonction : Auteur

Résumé

Abstract The rising prevalence of autism spectrum disorder, coupled with limited professional resources, highlights the urgency of developing efficient diagnostic tools. While standardized assessments exist, identifying subtle communication deficits, especially during multimodal interactions, remains time-consuming and prone to human error. To address this, we propose an automated behavior analysis framework that aims to support clinicians by accurately detecting both verbal and non-verbal communication markers. Specifically, we put forth a composite artificial intelligence framework that integrates various deep learning algorithms to analyze information from body and hand poses, object detection, tracking and manipulation, and speech. By combining these features with a rule-based system, we can identify events within the Autism Diagnostic Observation Schedule second edition, Construction Task, where participants initiate requests. These requests can be verbal, non-verbal or a combination of both resulting in multimodal interactions. Building on our prior work, this paper introduces a smart glass technology component, integrating gaze and blinking analysis, which are challenging for clinicians to monitor, given the multi-task nature of their role. These additions enable the detection of eye contact, a crucial social cue. Our approach allows us to recognize gestures, identify hand object manipulations, detect eye contact, and understand the natural language in clinician-participant interactions. We achieve 94% and 73% F -1 score, on verbal and non-verbal request detection, respectively, which may improve, as deep learning advances.

Fichier non déposé

Dates et versions

hal-05212410 , version 1 (18-08-2025)

Identifiants

Citer

Bruno Carlos dos Santos Melício, Kaan Karaköse, Ádám Fodor, Linyun Xiang, Viktor Varga, et al.. Multimodal Framework for Automatic Behavior Analysis of Children with Autism During ADOS-2. Cognitive Computation, 2025, 17 (4), pp.137. ⟨10.1007/s12559-025-10481-7⟩. ⟨hal-05212410⟩
150 Consultations
0 Téléchargements

Altmetric

Partager

  • More