Statistically valid links and anti-links between words and between documents: applying TourneBool randomization test to a Reuters collection. - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Advances in Knowledge Discovery and Management (AKDM) Année : 2010

Statistically valid links and anti-links between words and between documents: applying TourneBool randomization test to a Reuters collection.

Résumé

Neighborhood is a central concept in data mining, and a bunch of definitions have been implemented, mainly rooted in geometrical or topological considerations. We propose here a statistical definition of neighborhood: our TourneBool randomization test processes an objects $\times$ attributes binary table in order to establish which inter-attribute relations are fortuitous, and which ones are meaningful, without requiring any pre-defined statistical model, while taking into account the empirical distributions. It ensues a robust and statistically validated graph. We present a full-scale experiment on one of the public access Reuters test corpus. We characterize the resulting word graph by a series of indicators, such as clustering coefficients, degree distribution and correlation, cluster modularity and size distribution. Another graph structure stems from this process: the one conveying the negative ``counter-relations'' between words, i.e. words which ``steer clear'' one from another. We characterize in the same way the counter-relation graph. At last we generate the couple of valid document graphs (i.e. links and anti-links) and evaluate them by taking into account the Reuters document categories.

Dates et versions

hal-00429434 , version 1 (02-11-2009)

Identifiants

Citer

Alain Lelu, Martine Cadot. Statistically valid links and anti-links between words and between documents: applying TourneBool randomization test to a Reuters collection.. Advances in Knowledge Discovery and Management (AKDM), 2010, 292, pp.307-324. ⟨10.1007/978-3-642-00580-0⟩. ⟨hal-00429434⟩
128 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More