SET, SORT! A Novel Sub-Stroke Level Transformers for Offline Handwriting to Online Conversion - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

SET, SORT! A Novel Sub-Stroke Level Transformers for Offline Handwriting to Online Conversion

Résumé

We present a novel sub-stroke level transformer approach to convert offline images of handwriting to online. We start by extracting sub-strokes from the offline images by inferring a skeleton with a CNN and applying a basic cutting algorithm. We introduce sub-stroke embeddings by encoding the sub-stroke point sequence with a Sub-stroke Encoding Transformer (SET). The embeddings are then fed to the Substrokes ORdering Transformer (SORT) which predicts the discrete substrokes ordering and the pen state. By constraining the Transformer input and output to the inferred sub-strokes, the recovered online is highly precise. We evaluate our method on Latin words from the IRONOFF dataset and on maths expressions from CROHME dataset. We measure the performance with two criteria: fidelity with Dynamic Time Warping (DTW) and semantic coherence using recognition rate. Our method outperforms the state-of-the-art in both datasets, achieving a word recognition rate of 81.06% and a 2.41 DTW on IRONOFF and an expression recognition rate of 62.00% and a DTW of 13.93 on CROHME 2019. This work constitutes an important milestone toward full offline document conversion to online.
Fichier principal
Vignette du fichier
main.pdf (7.57 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04182547 , version 1 (17-08-2023)

Identifiants

Citer

Elmokhtar Mohamed Moussa, Thibault Lelore, Harold Mouchère. SET, SORT! A Novel Sub-Stroke Level Transformers for Offline Handwriting to Online Conversion. The 17th International Conference on Document Analysis and Recognition (ICDAR 2023), Aug 2023, San José, CA, United States. pp.81-97, ⟨10.1007/978-3-031-41676-7_5⟩. ⟨hal-04182547⟩
37 Consultations
42 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More