Text segmentation: a topic modeling perspective - Archive ouverte HAL
Article Dans Une Revue Information Processing and Management Année : 2011

Text segmentation: a topic modeling perspective

Résumé

In this paper, the task of text segmentation is approached from a topic modeling perspective. We investigate the use of two unsupervised topic models, latent Dirichlet allocation (LDA) and multinomial mixture (MM), to segment a text into semantically coherent parts. The proposed topic model based approaches consistently outperform a standard baseline method on several datasets. A major benefit of the proposed LDA based approach is that along with the segment boundaries, it outputs the topic distribution associated with each segment. This information is of potential use in applications such as segment retrieval and discourse analysis. However, the proposed approaches, especially the LDA based method, have high computational requirements. Based on an analysis of the dynamic programming (DP) algorithm typically used for segmentation, we suggest a modification to DP that dramatically speeds up the process with no loss in performance. The proposed modification to the DP algorithm is not specific to the topic models only; it is applicable to all the algorithms that use DP for the task of text segmentation.
Fichier principal
Vignette du fichier
Misra2011.pdf (229.67 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01960703 , version 1 (17-07-2019)

Identifiants

Citer

Hemant Misra, François Yvon, Jaemon Jose, Olivier Cappé. Text segmentation: a topic modeling perspective. Information Processing and Management, 2011, 47, pp.528-544. ⟨10.1016/j.ipm.2010.11.008⟩. ⟨hal-01960703⟩
97 Consultations
2045 Téléchargements

Altmetric

Partager

More