QAlayout: Question Answering Layout based on multimodal Attention for visual question answering on corporate Document - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Acoustics Research Letters Online Année : 2022

QAlayout: Question Answering Layout based on multimodal Attention for visual question answering on corporate Document

Résumé

The extraction of information from corporate documents is increasing in the research field both for its economic aspect and a scientific challenge. To extract this information the use of textual and visual content becomes unavoidable to understand the inherent information of the image. The information to be extracted is most often fixed beforehand (i.e. classification of words by date, total amount, etc). The information to be extracted is evolving, so we would not like to be restricted to predefine word classes. We would like to question a document such as "which is the address of invoicing?" as we can have several addresses in an invoice. We formulate our request as a question and our model will try to answer. Our model got the result 77.65% on the Docvqa dataset while drastically reducing the number of model parameters to allow us to use it in an industrial context and we use an attention model using several modalities that help us in the interpertation of the results obtained. Our other contribution in this paper is a new dataset for Visual Question answering on corporate document of invoices from RVL-CDIP[8]. The public data on corporate documents are less present in the state-of-theart, this contribution allow us to test our models to the invoice data with the VQA methods.
Fichier principal
Vignette du fichier
paper34_ibrahim_Souleiman.pdf (1.34 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04089332 , version 1 (24-05-2023)

Licence

Paternité - Pas d'utilisation commerciale - Pas de modification

Identifiants

  • HAL Id : hal-04089332 , version 1

Citer

Ibrahim Souleiman Mahamoud, Mickaël Coustaty, Aurélie Joseph, Vincent Poulain d'Andecy, Jean-Marc Ogier. QAlayout: Question Answering Layout based on multimodal Attention for visual question answering on corporate Document. Acoustics Research Letters Online, 2022. ⟨hal-04089332⟩

Collections

UNIV-ROCHELLE
6 Consultations
129 Téléchargements

Partager

Gmail Facebook X LinkedIn More