The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2013

The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction

Résumé

In this paper, we examine notions of text quality in the context of web corpus construction. Web documents often contain material which disqualifies them from inclusion in a corpus (tag clouds, lists of names or nouns, etc.). First, we look at the agreement between coders (especially corpus designers) given the task of rating text quality. Then, we evaluate a simple and fully unsupervised method of text quality assessment based on short and very frequent words. Finally, we describe our general approach to the construction of carefully cleansed and non-destructively normalized web corpora. Under this approach, we annotate documents with quality metrics instead of actually removing those documents classified as being of low quality.
Fichier principal
Vignette du fichier
SchaeferBarbaresiBildhauer_GoodBadHazyDesignDecisionsWebCorpora_Wac8_2013.pdf (5.23 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01091602 , version 1 (05-12-2014)

Identifiants

  • HAL Id : hal-01091602 , version 1

Citer

Roland Schäfer, Adrien Barbaresi, Felix Bildhauer. The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction. 8th Web as Corpus Workshop, ACL SIGWAC, Jul 2013, Lancaster, United Kingdom. pp.7-15. ⟨hal-01091602⟩
197 Consultations
137 Téléchargements

Partager

Gmail Facebook X LinkedIn More