Focused Web Corpus Crawling - Archive ouverte HAL
Communication Dans Un Congrès Année : 2014

Focused Web Corpus Crawling

Résumé

In web corpus construction, crawling is a necessary step, and it is probably the most costly of all, because it requires expensive bandwidth usage, and excess crawling increases storage requirements. Excess crawling results from the fact that the web contains a lot of redundant content (duplicates and near-duplicates), as well as other material not suitable or desirable for inclusion in web corpora or web indexes (for example, pages with little text or virtually no text at all). An optimized crawler for web corpus construction would ideally avoid crawling such content in the first place, saving bandwidth, storage, and post-processing costs. In this paper, we show in three experiments that two simple scores are suitable to improve the ratio between corpus size and crawling effort for web corpus construction. The first score is related to overall text quality of the page containing the link, the other one is related to the likelihood that the local block enclosing a link is boilerplate.

Domaines

Linguistique
Fichier principal
Vignette du fichier
Schaefer-Barbaresi-Bildhauer_WAC-9_2014.pdf (339.2 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01095924 , version 1 (16-12-2014)

Identifiants

  • HAL Id : hal-01095924 , version 1

Citer

Roland Schäfer, Adrien Barbaresi, Felix Bildhauer. Focused Web Corpus Crawling. 9th Web as Corpus Workshop (WaC-9), Apr 2014, Gothenburg, Sweden. pp.9-15. ⟨hal-01095924⟩
258 Consultations
308 Téléchargements

Partager

More