Hierarchical and Multimodal Classification of Images from Soil Remediation Reports - Archive ouverte HAL Accéder directement au contenu
Chapitre D'ouvrage Année : 2021

Hierarchical and Multimodal Classification of Images from Soil Remediation Reports

Résumé

When soil remediation specialists clean up a new site, they have a long time manually revising digital reports previously written by other experts, where they look for necessary information in accordance with similar characteristics of polluted fields. Important information lies in tables, graphs, maps, drawings and their associated captions. Therefore, experts have to be able to quickly access these content-rich elements, instead of manually scrolling through each page of entire reports. Since this information is multimodal (image and text) and follows a semantically hierarchical structure, we propose a classification algorithm that takes these two constraints into account. In contrast to existing works using either multimodal system or hierarchical classification model, we explore the combination of state-of-the-art methods from multimodal systems (image and text modalities) and hierarchical classification systems. By this combination, we tackle the constraints of our classification process: small dataset, missing modalities, noisy data, and non-English corpus. Our evaluation shows that the multimodal hierarchical system outperforms the unimodal and that the performance of multimodal system with a joint combination of hierarchical classification and flat classification on different modalities provides promising results.
Fichier principal
Vignette du fichier
Hierarchical and Multimodal Classification of Images from Soil Remediation reports_hal.pdf (849.23 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03360311 , version 1 (30-09-2021)

Identifiants

Citer

Korlan Rysbayeva, Romain Giot, Nicholas Journet. Hierarchical and Multimodal Classification of Images from Soil Remediation Reports. Document Analysis and Recognition – ICDAR 2021, 12821, Springer International Publishing, pp.160-175, 2021, Lecture Notes in Computer Science, ⟨10.1007/978-3-030-86549-8_11⟩. ⟨hal-03360311⟩

Collections

CNRS
37 Consultations
84 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More