Can we predict how challenging Spoken Language Understanding corpora are across sources, languages and domains? - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Can we predict how challenging Spoken Language Understanding corpora are across sources, languages and domains?

Résumé

State-of-the art Spoken Language Understanding models of Spoken Dialog Systems achieve remarkable results on benchmark corpora thanks to the winning combination of pretraining on large collection of out-of-domain data with contextual Transformer representations and fine tuning on in-domain data. On average, performances are almost perfect on benchmark datasets such ATIS. However some phenomena can affect greatly these performance, like unseen events or ambiguities. They are the major sources of errors in real-life deployed systems although they are not necessarily equally represented in benchmark corpora. This paper aims to predict and characterize error-prone utterances and to explain what makes a given corpus more or less challenging. After training such a predictor on benchmark corpora from various languages and domains, we confront it to a new corpus collected from a French deployed vocal assistant with different distributional properties. We show that the predictor can highlight challenging utterances and explain the main complexity factors even though this corpus was collected in a completely different setting.
Fichier principal
Vignette du fichier
IWSDS2021 (12).pdf (141.5 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03386025 , version 1 (19-10-2021)

Identifiants

  • HAL Id : hal-03386025 , version 1

Citer

Frederic Bechet, Christian Raymond, Achraf Hamane, Rim Abrougui, Gabriel Marzinotto, et al.. Can we predict how challenging Spoken Language Understanding corpora are across sources, languages and domains?. The 12th International Workshop on Spoken Dialog System Technology, Nov 2021, Singapore, Singapore. ⟨hal-03386025⟩
136 Consultations
208 Téléchargements

Partager

Gmail Facebook X LinkedIn More