Communication Dans Un Congrès Année : 2021

Manga-MMTL: Multimodal Multitask Transfer Learning for Manga Character Analysis

Résumé

In this paper, we introduce a new pipeline to learn manga character features with visual information and verbal information in manga image content. Combining these set of information is crucial to go further into comic book image understanding. However, learning feature representations from multiple modalities is not straightforward. We propose a multitask multimodal approach for effectively learning the feature of joint multimodal signals. To better leverage the verbal information, our method learn to memorize the content of manga albums by additionally using the album classification task. The experiments are carried out on Manga109 public dataset which contains the annotations for characters, text blocks, frame and album metadata. We show that manga character features learnt by the proposed method is better than all existing single-modal methods for two manga character analysis tasks.

Fichier principal
Vignette du fichier
ICDAR2021_MangaMMTL.pdf (328.6 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04228616 , version 1 (04-10-2023)

Licence

Identifiants

Citer

Nhu-Van Nguyen, Christophe Rigaud, Arnaud Revel, Jean-Christophe Burie. Manga-MMTL: Multimodal Multitask Transfer Learning for Manga Character Analysis. International Conference on Document Analysis and Recognition, Sep 2021, Lausanne, Switzerland. pp.410-425, ⟨10.1007/978-3-030-86331-9_27⟩. ⟨hal-04228616⟩

Collections

98 Consultations
639 Téléchargements

Altmetric

Partager

  • More