A Survey on Handwritten Mathematical Expression Recognition: The Rise of Encoder-Decoder and GNN Models - Archive ouverte HAL
Article Dans Une Revue Pattern Recognition Année : 2024

A Survey on Handwritten Mathematical Expression Recognition: The Rise of Encoder-Decoder and GNN Models

Résumé

Recognition of handwritten mathematical expressions (HMEs) has attracted growing interest due to steady progress in handwriting recognition techniques and the rapid emergence of pen- and touch-based devices. Math formula recognition may be understood as a generalization of text recognition: formulas represent mathematical statements using a two dimensional arrangement of symbols on writing lines that are organized hierarchically. This survey provides an overview of techniques published in the last decade, including those taking input from handwritten strokes (i.e., ‘online’, as captured by a pen/touch device), raster images (i.e., ‘offline,’ from pixels), or both. Traditionally, HMEs were recognized by performing four structural pattern recognition tasks in separate steps: (1) symbol segmentation, (2) symbol classification, (3) spatial relationship classification, and (4) structural analysis, which identifies the arrangement of symbols on writing lines (e.g., in a Symbol Layout Tree (SLT) or LaTeX string). Recently, encoder–decoder neural network models and Graph Neural Network (GNN) approaches have greatly increased HME recognition accuracy. These newer approaches perform all recognition tasks simultaneously, and utilize contextual features across tasks (e.g., using neural self-attention models). We also discuss evaluation techniques and benchmarks, and explore some implicit dependencies among the four key recognition tasks. Finally, we identify limitations of current systems, and present suggestions for future work, such as using two-dimensional language models rather than the one-dimensional models commonly used in encoder–decoder models.
Fichier principal
Vignette du fichier
HME_survey_in_this_decade_vf.pdf (873.9 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04560379 , version 1 (26-04-2024)

Identifiants

Citer

Thanh-Nghia Truong, Cuong Tuan Nguyen, Richard Zanibbi, Harold Mouchère, Masaki Nakagawa. A Survey on Handwritten Mathematical Expression Recognition: The Rise of Encoder-Decoder and GNN Models. Pattern Recognition, In press, ⟨10.1016/j.patcog.2024.110531⟩. ⟨hal-04560379⟩
168 Consultations
379 Téléchargements

Altmetric

Partager

More