A task-oriented framework for evaluating theme detection systems. A discussion paper. - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2006

A task-oriented framework for evaluating theme detection systems. A discussion paper.

Résumé

This paper discusses the inherent difficulties in evaluating systems for theme detection. Such systems are based essentially on unsupervised clustering aiming to discover the underlying structure in a corpus of texts. As the structures are precisely unknown beforehand, it is difficult to devise a satisfactory evaluation protocol. Several problems are posed by cluster evaluation: determining the optimal number of clusters, cluster content evaluation, topology of the discovered structure. Each of these problems has been studied separately but some of the proposed metrics portray significant flaws. Moreover, no benchmark has been commonly agreed upon. Finally, it is necessary to distinguish between task-oriented and activity-oriented evaluation as the two frameworks imply different evaluation protocols. Possible solutions to the activity-oriented evaluation can be sought from the data and text mining communities.
Fichier principal
Vignette du fichier
ibekwe-LREC06.pdf (179.29 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00636148 , version 1 (26-10-2011)

Identifiants

  • HAL Id : hal-00636148 , version 1

Citer

Fidelia Ibekwe-Sanjuan. A task-oriented framework for evaluating theme detection systems. A discussion paper.. 5th International Conference on Language Resources and Evaluation Conference (LREC-06)., May 2006, Genoa, Italy. pp.2492-2497. ⟨hal-00636148⟩
128 Consultations
55 Téléchargements

Partager

Gmail Facebook X LinkedIn More