Improving End-to-end Sign Language Translation with Adaptive Video Representation Enhanced Transformer - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Circuits and Systems for Video Technology Année : 2024

Improving End-to-end Sign Language Translation with Adaptive Video Representation Enhanced Transformer

Résumé

The aim of end-to-end sign language translation (SLT) is to interpret continuous sign language (SL) video sequences into coherent natural language sentences without any intermediary annotations, i.e., glosses. However, end-to-end SLT suffers several intractable issues: (i) the temporal correspondence constraint loss problem between SL videos and glosses, and (ii) the weakly supervised sequence labeling problem between long SL videos and sentences. To address these issues, we propose an adaptive video representation enhanced Transformer (AVRET), with three extra modules: adaptive masking (AM), local clip self-attention (LCSA) and adaptive fusion (AF). Specifically, we utilize the first AM module to generate a special mask that adaptively drops out temporally important SL video frame representations to enhance the SL video features. Then, we pass the masked video feature to the Transformer encoder consisting of LCSA and masked self-attention to learn clip-level and continuous video-level feature information. Finally, the output feature of encoder is fused with the temporal feature of AM module via the AF module and use the second AM module to generate more robust feature representations. Besides, we add weakly supervised loss terms to constrain these two AM modules. To promote the Chinese SLT research, we further construct CSL-FocusOn, a Chinese continuous SLT dataset, and share its collection method. It involves many common scenarios, and provides SL sentence annotations and multi-cue images of signers. Our experiments on the CSL-FocusOn, PHOENIX14T, and CSL-Daily datasets show that the proposed method achieves the competitive performance on the end-to-end SLT task without using glosses in training. The code is available at https://github.com/LzDddd/AVRET.
Fichier principal
Vignette du fichier
Liu et al_2024_Improving End-to-end Sign Language Translation with Adaptive Video representation enhanced transformer.pdf (1.74 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY NC - Paternité - Pas d'utilisation commerciale

Dates et versions

hal-04526587 , version 1 (23-05-2024)

Licence

Paternité - Pas d'utilisation commerciale

Identifiants

Citer

Zidong Liu, Jiasong Wu, Zeyu Shen, Xin Chen, Qianyu Wu, et al.. Improving End-to-end Sign Language Translation with Adaptive Video Representation Enhanced Transformer. IEEE Transactions on Circuits and Systems for Video Technology, 2024, pp.1-1. ⟨10.1109/tcsvt.2024.3376404⟩. ⟨hal-04526587⟩
6 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More